zeppelin_notebook.json

{"paragraphs":[{"text":"%md\n\n# Annual Revenue Vs. Executive Pay \n## for \n# Recipients of U.S. Federal Funds\n\n\n**Data Source:** USASpending.gov\n\nAccording to their website: USAspending.gov is the publicly accessible, searchable website mandated by the Federal Funding Accountability and Transparency Act of 2006 to give the American public access to information on how their tax dollars are spent.","user":"anonymous","dateUpdated":"2017-08-24T18:03:36+0000","config":{"colWidth":12,"editorMode":"ace/mode/markdown","results":[{"graph":{"mode":"table","height":300,"optionOpen":false,"keys":[],"values":[],"groups":[],"scatter":{}}}],"enabled":true,"editorSetting":{"language":"markdown","editOnDblClick":true},"editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<h1>Annual Revenue Vs. Executive Pay</h1>\n<h2>for</h2>\n<h1>Recipients of U.S. Federal Funds</h1>\n<p><strong>Data Source:</strong> USASpending.gov</p>\n<p>According to their website: USAspending.gov is the publicly accessible, searchable website mandated by the Federal Funding Accountability and Transparency Act of 2006 to give the American public access to information on how their tax dollars are spent.</p>\n"}]},"apps":[],"jobName":"paragraph_1503460484143_-294962124","id":"20160730-190636_801628602","dateCreated":"2017-08-23T03:54:44+0000","dateStarted":"2017-08-24T18:03:36+0000","dateFinished":"2017-08-24T18:03:36+0000","status":"FINISHED","progressUpdateIntervalMs":500,"focus":true,"$$hashKey":"object:7221"},{"text":"%md\n\n#### Data extraction\n\nThe data was downloaded as a batched CSV file from USASpending.gov. The CSV file contained a record of all federal funding allocation to companies and non-profits in 2015.\n\nOf the 225 fields in the original data, we are most interested in dollarsobligated, vendorname, annualrevenue, prime_awardee_executive1_compensation, prime_awardee_executive2_compensation, prime_awardee_executive3_compensation, prime_awardee_executive4_compensation, and prime_awardee_executive5_compensation.\n\nThe original CSV file had 261 line breaks within individual data fields out of over 20M rows of data. The CSV could not be loaded into a Spark Dataframe with this inconsistent formatting. The problematic rows were removed from the data set for this exercise.\n\nHere are the steps I took to clean the data from the UNIX command line:\n\n1. Select for lines that begin and end with a quote.\n\n     pv Data_Feed.csv | grep \"^\\\".*\\\"\\r\" > data_feed_good.csv\n    \n    \n2. Filter out lines that begin with an end-quote followed by a comma.\n\n     pv data_feed_good.csv | grep -v \"^\\\",.*\\r\" > data_feed_good_2.csv \n\n**Note:**  These data cleansing steps remove 0.001305% of data.\n\n\nNow, let's read the cleaned parquet file from S3 to a Spark data frame.","user":"anonymous","dateUpdated":"2017-08-24T17:59:42+0000","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<h4>Data extraction</h4>\n<p>The data was downloaded as a batched CSV file from USASpending.gov. The CSV file contained a record of all federal funding allocation to companies and non-profits in 2015.</p>\n<p>Of the 225 fields in the original data, we are most interested in dollarsobligated, vendorname, annualrevenue, prime_awardee_executive1_compensation, prime_awardee_executive2_compensation, prime_awardee_executive3_compensation, prime_awardee_executive4_compensation, and prime_awardee_executive5_compensation.</p>\n<p>The original CSV file had 261 line breaks within individual data fields out of over 20M rows of data. The CSV could not be loaded into a Spark Dataframe with this inconsistent formatting. The problematic rows were removed from the data set for this exercise.</p>\n<p>Here are the steps I took to clean the data from the UNIX command line:</p>\n<ol>\n<li><p>Select for lines that begin and end with a quote.</p>\n<p>pv Data_Feed.csv | grep &ldquo;^\".*\"\\r&rdquo; > data_feed_good.csv</p>\n</li>\n<li><p>Filter out lines that begin with an end-quote followed by a comma.</p>\n<p>pv data_feed_good.csv | grep -v &ldquo;^\",.*\\r&rdquo; > data_feed_good_2.csv</p>\n</li>\n</ol>\n<p><strong>Note:</strong>  These data cleansing steps remove 0.001305% of data.</p>\n<p>Now, let's read the cleaned parquet file from S3 to a Spark data frame.</p>\n"}]},"apps":[],"jobName":"paragraph_1503519274065_338714619","id":"20170823-201434_1974545042","dateCreated":"2017-08-23T20:14:34+0000","dateStarted":"2017-08-24T17:59:42+0000","dateFinished":"2017-08-24T17:59:42+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:7222"},{"text":"val df = sqlContext.read.parquet(\"s3://sarah-usaspendingfy2015/clean_data.parquet\")","user":"anonymous","dateUpdated":"2017-08-24T17:58:41+0000","config":{"colWidth":12,"editorMode":"ace/mode/scala","results":[{"graph":{"mode":"table","height":300,"optionOpen":false,"keys":[],"values":[],"groups":[],"scatter":{}}}],"enabled":true,"editorSetting":{"language":"scala","editOnDblClick":false}},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"\ndf: org.apache.spark.sql.DataFrame = [unique_transaction_id: string, transaction_status: string ... 223 more fields]\n"}]},"apps":[],"jobName":"paragraph_1503460484145_-284958652","id":"20160730-170035_443062780","dateCreated":"2017-08-23T03:54:44+0000","dateStarted":"2017-08-24T17:58:41+0000","dateFinished":"2017-08-24T17:58:49+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:7223"},{"text":"df.columns","user":"anonymous","dateUpdated":"2017-08-24T17:58:41+0000","config":{"colWidth":12,"enabled":true,"results":{"0":{"graph":{"mode":"table","height":91,"optionOpen":false}}},"editorSetting":{"language":"scala","editOnDblClick":false},"editorMode":"ace/mode/scala","tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"res235: Array[String] = Array(unique_transaction_id, transaction_status, dollarsobligated, baseandexercisedoptionsvalue, baseandalloptionsvalue, maj_agency_cat, mod_agency, maj_fund_agency_cat, contractingofficeagencyid, contractingofficeid, fundingrequestingagencyid, fundingrequestingofficeid, fundedbyforeignentity, signeddate, effectivedate, currentcompletiondate, ultimatecompletiondate, lastdatetoorder, contractactiontype, reasonformodification, typeofcontractpricing, priceevaluationpercentdifference, subcontractplan, lettercontract, multiyearcontract, performancebasedservicecontract, majorprogramcode, contingencyhumanitarianpeacekeepingoperation, contractfinancing, costorpricingdata, costaccountingstandardsclause, descriptionofcontractrequirement, purchasecardaspaymentmethod, number..."}]},"apps":[],"jobName":"paragraph_1503519229278_-1452938416","id":"20170823-201349_1197785319","dateCreated":"2017-08-23T20:13:49+0000","dateStarted":"2017-08-24T17:58:41+0000","dateFinished":"2017-08-24T17:58:49+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:7224"},{"text":"%md\n\n#### How many contracts, loans, and grants were obligated by the federal government in 2015?","user":"anonymous","dateUpdated":"2017-08-24T17:59:51+0000","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<h4>How many contracts, loans, and grants were obligated by the federal government in 2015?</h4>\n"}]},"apps":[],"jobName":"paragraph_1503521032877_1863771126","id":"20170823-204352_2092496706","dateCreated":"2017-08-23T20:43:52+0000","dateStarted":"2017-08-24T17:59:51+0000","dateFinished":"2017-08-24T17:59:51+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:7225"},{"text":"df.count","user":"anonymous","dateUpdated":"2017-08-24T17:58:41+0000","config":{"colWidth":12,"editorMode":"ace/mode/scala","results":[{"graph":{"mode":"table","height":300,"optionOpen":false,"keys":[],"values":[],"groups":[],"scatter":{}}}],"enabled":true,"editorSetting":{"language":"scala","editOnDblClick":false}},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"\nres236: Long = 3679400\n"}]},"apps":[],"jobName":"paragraph_1503460484146_-283804406","id":"20160730-182308_1097299057","dateCreated":"2017-08-23T03:54:44+0000","dateStarted":"2017-08-24T17:58:49+0000","dateFinished":"2017-08-24T17:58:52+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:7226"},{"text":"%md \n\n#### What companies have the most federal dollars obligated to them?","user":"anonymous","dateUpdated":"2017-08-24T17:59:54+0000","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<h4>What companies have the most federal dollars obligated to them?</h4>\n"}]},"apps":[],"jobName":"paragraph_1503519626827_1653461786","id":"20170823-202026_1322156220","dateCreated":"2017-08-23T20:20:26+0000","dateStarted":"2017-08-24T17:59:54+0000","dateFinished":"2017-08-24T17:59:54+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:7227"},{"text":"val dollarsObligated = df.select(df(\"vendorname\"), df(\"dollarsobligated\"))\n\t.groupBy(\"vendorname\")\n\t.sum(\"dollarsobligated\")\n\t.sort($\"sum(dollarsobligated)\".desc);\ndollarsObligated.show()\n","user":"anonymous","dateUpdated":"2017-08-24T17:58:49+0000","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"scala","editOnDblClick":false},"editorMode":"ace/mode/scala"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"\ndollarsObligated: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] = [vendorname: string, sum(dollarsobligated): double]\n+--------------------+---------------------+\n|          vendorname|sum(dollarsobligated)|\n+--------------------+---------------------+\n|LOCKHEED MARTIN C...| 2.485973147905999...|\n|    RAYTHEON COMPANY|  8.666833321250006E9|\n| BOEING COMPANY, THE|  7.604776054149998E9|\n|MCKESSON CORPORATION|  7.563494564229999E9|\n|NORTHROP GRUMMAN ...|  6.322189713929999E9|\n|ELECTRIC BOAT COR...|      5.02475097855E9|\n|  THE BOEING COMPANY|      4.90264890895E9|\n|SCIENCE APPLICATI...| 3.4890949293700027E9|\n|HUMANA MILITARY H...|  3.313841091250001E9|\n|HUNTINGTON INGALL...|      3.24046953884E9|\n|SIKORSKY AIRCRAFT...| 3.1885837808500004E9|\n|UNITED TECHNOLOGI...|      2.92949119238E9|\n|L-3 COMMUNICATION...|      2.86854365592E9|\n|         SANDIA CORP|      2.80871100978E9|\n|HEALTH NET FEDERA...| 2.7825438602300005E9|\n|BECHTEL MARINE PR...|      2.70279553093E9|\n|UNITEDHEALTH MILI...|      2.50534454686E9|\n|UNITED LAUNCH SER...|      2.09988312678E9|\n|BOOZ ALLEN HAMILT...| 2.0912978022100005E9|\n|LOS ALAMOS NATION...| 2.0885016060100002E9|\n+--------------------+---------------------+\nonly showing top 20 rows\n\n"}]},"apps":[],"jobName":"paragraph_1503506782495_556135939","id":"20170823-164622_1499464446","dateCreated":"2017-08-23T16:46:22+0000","dateStarted":"2017-08-24T17:58:49+0000","dateFinished":"2017-08-24T17:59:00+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:7228"},{"text":"%md\n\n#### What companies have the highest self-reported annual revenue?","user":"anonymous","dateUpdated":"2017-08-24T17:58:42+0000","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<h4>What companies have the highest self-reported annual revenue?</h4>\n"}]},"apps":[],"jobName":"paragraph_1503519599414_-2050946597","id":"20170823-201959_1561026085","dateCreated":"2017-08-23T20:19:59+0000","dateStarted":"2017-08-24T17:58:42+0000","dateFinished":"2017-08-24T17:58:42+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:7229"},{"text":"df.select(\n    $\"vendorname\", \n    $\"annualrevenue\")\n    .sort( $\"annualrevenue\".desc)\n    .distinct().show()\n","user":"anonymous","dateUpdated":"2017-08-24T17:58:52+0000","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"scala","editOnDblClick":false},"editorMode":"ace/mode/scala"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"+--------------------+---------------+\n|          vendorname|  annualrevenue|\n+--------------------+---------------+\n|SIERRA NEVADA COR...|937973411479552|\n|   XEROX CORPORATION|179999999000576|\n|GEOLOGICAL & ECON...|155529003401216|\n|D & L WASTE EQUIP...| 99999999999999|\n|          APPLE INC.| 46900000000000|\n|          APPLE INC.| 46899998490624|\n|METAL-FLEX WELDED...| 31582003920896|\n|          APPLE INC.| 25600000000000|\n|          APPLE INC.| 25599999475712|\n|VERIZON BUSINESS ...| 25599999475712|\n|       DRS ICAS, LLC| 25000000000000|\n|BAE SYSTEMS LAND ...| 24613600000000|\n|BAE SYSTEMS LAND ...| 24613599838208|\n|      HEDGEROW FARMS| 21126761676800|\n|          APPLE INC.| 19999999655936|\n|BRIDGESTONE RETAI...| 13205000000000|\n|BRIDGESTONE RETAI...| 13204999634944|\n|CONSOLIDATED EDIS...|  9885000400896|\n|KOREA RAILROAD CO...|  8640813596672|\n|KOREA RAILROAD CO...|  8640813353029|\n+--------------------+---------------+\nonly showing top 20 rows\n\n"}]},"apps":[],"jobName":"paragraph_1503506908334_916092343","id":"20170823-164828_703627047","dateCreated":"2017-08-23T16:48:28+0000","dateStarted":"2017-08-24T17:58:52+0000","dateFinished":"2017-08-24T17:59:04+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:7230"},{"text":"%md\n\n#### What companies have the highest executive compensation (as reported by max of executive compensation fields)?\n\nThe highest executive compensation reported is $950 Bilion by SCHAFER AEROSPACE.","user":"anonymous","dateUpdated":"2017-08-24T17:58:42+0000","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<h4>What companies have the highest executive compensation (as reported by max of executive compensation fields)?</h4>\n<p>The highest executive compensation reported is $950 Bilion by SCHAFER AEROSPACE.</p>\n"}]},"apps":[],"jobName":"paragraph_1503519562876_-1830886074","id":"20170823-201922_311244806","dateCreated":"2017-08-23T20:19:22+0000","dateStarted":"2017-08-24T17:58:42+0000","dateFinished":"2017-08-24T17:58:42+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:7231"},{"text":"import scala.util.Try","user":"anonymous","dateUpdated":"2017-08-24T17:58:42+0000","config":{"colWidth":12,"graph":{"mode":"table","height":300,"optionOpen":false,"keys":[],"values":[],"groups":[],"scatter":{}},"enabled":true,"results":{},"editorSetting":{"language":"scala","editOnDblClick":false},"editorMode":"ace/mode/scala"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"\nimport scala.util.Try\n"}]},"apps":[],"jobName":"paragraph_1503460484145_-284958652","id":"20160730-190623_2133172018","dateCreated":"2017-08-23T03:54:44+0000","dateStarted":"2017-08-24T17:59:00+0000","dateFinished":"2017-08-24T17:59:04+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:7232"},{"text":"val execCompensation = df.select(\r    $\"vendorname\", \r    $\"prime_awardee_executive1_compensation\", \r    $\"prime_awardee_executive2_compensation\", \r    $\"prime_awardee_executive3_compensation\", \r    $\"prime_awardee_executive4_compensation\",\r    $\"prime_awardee_executive5_compensation\")\r    .rdd\r    .map(row => {\r        val vendorName: String = row.getString(0); \r        val maxExecCompensation: Option[Double] = \r            Try(Some((1 to 5).map(n => \r                Try(row.getDouble(n))\r                .getOrElse(0.0)).max)).getOrElse(None);\r        (vendorName, maxExecCompensation)\r    })\r    .toDF.sort(desc(\"_2\")).distinct()\r    .withColumnRenamed(\"_1\", \"vendorName\")\r    .withColumnRenamed(\"_2\", \"maxExecCompensation\")","user":"anonymous","dateUpdated":"2017-08-24T17:59:04+0000","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"scala","editOnDblClick":false},"editorMode":"ace/mode/scala"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"\nexecCompensation: org.apache.spark.sql.DataFrame = [vendorName: string, maxExecCompensation: double]\n"}]},"apps":[],"jobName":"paragraph_1503509738214_-65891198","id":"20170823-173538_1337946235","dateCreated":"2017-08-23T17:35:38+0000","dateStarted":"2017-08-24T17:59:04+0000","dateFinished":"2017-08-24T17:59:05+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:7233"},{"text":"execCompensation.show()","user":"anonymous","dateUpdated":"2017-08-24T17:59:05+0000","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"scala","editOnDblClick":false},"editorMode":"ace/mode/scala"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"+--------------------+-------------------+\n|          vendorName|maxExecCompensation|\n+--------------------+-------------------+\n|SCHAFER AEROSPACE...|   9.53842289652E11|\n| SCHAFER CORPORATION|   9.53842289652E11|\n|RESEARCH ANALYSIS...|           1.704E11|\n|RESEARCH ANALYSIS...|           1.704E11|\n|FUTURE CARE HEALT...|       1.49900992E8|\n|    ASE DIRECT, INC.|         1.180767E8|\n|    A S E DIRECT LLC|         1.180767E8|\n|PARAGON TECHNICAL...|           1.0658E8|\n|DIXON GROUP, INC....|              1.0E8|\n|         FORCE 3 INC|        8.8277633E7|\n|       FORCE 3, INC.|        8.8277633E7|\n|FORCE 3 INCORPORATED|        8.8277633E7|\n|           ECCO GMBH|        7.4968679E7|\n|ENVIRONMENTAL CHE...|        6.5707594E7|\n|     AM GENERAL, LLC|        5.2454072E7|\n|ENVIRONMENTAL CHE...|        4.9559414E7|\n|ENVIRONMENTAL CHE...|        4.9559414E7|\n|ENVIRONMENTAL CHE...|        4.9559414E7|\n|CHARLES RIVER ANA...|         4.671253E7|\n|AXIOM RESOURCE MA...|        4.5750944E7|\n+--------------------+-------------------+\nonly showing top 20 rows\n\n"}]},"apps":[],"jobName":"paragraph_1503518423702_165153994","id":"20170823-200023_522901758","dateCreated":"2017-08-23T20:00:23+0000","dateStarted":"2017-08-24T17:59:05+0000","dateFinished":"2017-08-24T17:59:14+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:7234"},{"text":"%md\n\n#### What companies have the highest executive pay (as a percentage of total reported annual revenue)?","user":"anonymous","dateUpdated":"2017-08-24T18:00:04+0000","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<h4>What companies have the highest executive pay (as a percentage of total reported annual revenue)?</h4>\n"}]},"apps":[],"jobName":"paragraph_1503519888539_-144886866","id":"20170823-202448_1227125824","dateCreated":"2017-08-23T20:24:48+0000","dateStarted":"2017-08-24T18:00:04+0000","dateFinished":"2017-08-24T18:00:04+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:7235"},{"text":"val percentExecComp = df.select(\r    $\"vendorname\", \r    $\"prime_awardee_executive1_compensation\", \r    $\"prime_awardee_executive2_compensation\", \r    $\"prime_awardee_executive3_compensation\", \r    $\"prime_awardee_executive4_compensation\",\r    $\"prime_awardee_executive5_compensation\", \r    $\"annualrevenue\").rdd\r    .map(row => {\r        val vendorName: String = row.getString(0); \r        val maxExecCompensation: Option[Double] = \r            Try(Some((1 to 5).map(n => \r                Try(row.getDouble(n)).getOrElse(0.0)).max)).getOrElse(None);\r        val annualRevenue: Option[Double] = \r            Try(Some(row.getLong(6).toDouble)).getOrElse(None);\r        val percentageCompensation: Option[Double] = \r            Try(Some(maxExecCompensation.get/annualRevenue.get)).getOrElse(None);\r        (vendorName, maxExecCompensation, annualRevenue, percentageCompensation)\r    }).filter(row => row._3.exists(_ > 0))\r    .distinct\r    .toDF.sort($\"_4\".desc)\r    .withColumnRenamed(\"_1\", \"vendorName\")\r    .withColumnRenamed(\"_2\", \"maxExecCompensation\")\r    .withColumnRenamed(\"_3\", \"annualRevenue\")\r    .withColumnRenamed(\"_4\", \"percent\")\r    .distinct","user":"anonymous","dateUpdated":"2017-08-24T17:59:05+0000","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"scala","editOnDblClick":false},"editorMode":"ace/mode/scala"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"\npercentExecComp: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] = [vendorName: string, maxExecCompensation: double ... 2 more fields]\n"}]},"apps":[],"jobName":"paragraph_1503508363386_-1655653919","id":"20170823-171243_593893143","dateCreated":"2017-08-23T17:12:43+0000","dateStarted":"2017-08-24T17:59:05+0000","dateFinished":"2017-08-24T17:59:14+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:7236"},{"text":"%md\n\nSome companies pay out high amounts to executives, while reported siginificanlty lower annual revenue.","user":"anonymous","dateUpdated":"2017-08-24T17:58:42+0000","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<p>Some companies pay out high amounts to executives, while reported siginificanlty lower annual revenue.</p>\n"}]},"apps":[],"jobName":"paragraph_1503521286110_1850657371","id":"20170823-204806_923152150","dateCreated":"2017-08-23T20:48:06+0000","dateStarted":"2017-08-24T17:58:42+0000","dateFinished":"2017-08-24T17:58:42+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:7237"},{"text":"percentExecComp.show()","user":"anonymous","dateUpdated":"2017-08-24T17:59:14+0000","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"scala","editOnDblClick":false},"editorMode":"ace/mode/scala"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"+--------------------+-------------------+-------------+--------------------+\n|          vendorName|maxExecCompensation|annualRevenue|             percent|\n+--------------------+-------------------+-------------+--------------------+\n|GENERAL DYNAMICS ...|          1810115.0|    9000000.0| 0.20112388888888888|\n|    PD SYSTEMS, INC.|           229058.0|    1765000.0| 0.12977790368271955|\n|         KNWEBS INC.|           192000.0|    1950000.0| 0.09846153846153846|\n|PROSOURCE CONSULT...|           445000.0|    5000000.0|               0.089|\n|DANYA INTERNATION...|          5045608.0|  6.8066532E7| 0.07412759034058031|\n|M & M CONTRACT MO...|            36000.0|     500000.0|               0.072|\n|  J'S ASSOCIATES LLC|           163462.0|    2454904.0| 0.06658590315547981|\n|LIFE SCIENCE LOGI...|           480000.0|    9726812.0| 0.04934813174141744|\n|BRAINERD HELICOPT...|           220000.0|   1.163553E7|0.018907604552607402|\n|WIEDEMANN CONSULTING|           240000.0|        1.5E7|               0.016|\n|SCIENCE AND TECHN...|           809644.0|  5.2216524E7|0.015505513159014567|\n|STRATEGIC RESOURC...|           257030.0|       1.85E7|0.013893513513513513|\n|HERNDON PRODUCTS,...|           980914.0|  7.8336466E7| 0.01252180561732259|\n|MATHEMATICA POLIC...|           766000.0|    6.60175E7|0.011602984057257546|\n|    PRAGMATICS, INC.|           746459.0|       6.61E7|0.011292874432677761|\n|INGENESIS ARORA S...|           259405.0|  4.2977472E7|0.006035836635528493|\n| HYDROGEOLOGIC, INC.|           389625.0|  6.8536808E7|0.005684901461999806|\n|  ROBBINS-GIOIA, LLC|           330357.0|  6.6474664E7|0.004969667842172169|\n|      FIBERTEK, INC.|           356326.0|  8.4083792E7|0.004237748935014729|\n|DELL SERVICES FED...|          1050420.0|       2.75E8|0.003819709090909...|\n+--------------------+-------------------+-------------+--------------------+\nonly showing top 20 rows\n\n"}]},"apps":[],"jobName":"paragraph_1503518448911_-857862683","id":"20170823-200048_122695524","dateCreated":"2017-08-23T20:00:48+0000","dateStarted":"2017-08-24T17:59:14+0000","dateFinished":"2017-08-24T17:59:22+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:7238"},{"text":"%md\n\n#### How many companies are there with annual revenue > $1M and executive compensation > $1M?","user":"anonymous","dateUpdated":"2017-08-24T17:58:42+0000","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<h4>How many companies are there with annual revenue > $1M and executive compensation > $1M?</h4>\n"}]},"apps":[],"jobName":"paragraph_1503523604623_1339988835","id":"20170823-212644_349185178","dateCreated":"2017-08-23T21:26:44+0000","dateStarted":"2017-08-24T17:58:42+0000","dateFinished":"2017-08-24T17:58:42+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:7239"},{"text":"(percentExecComp.count, percentExecComp.filter($\"maxExecCompensation\" > 1000000).filter($\"annualRevenue\"> 1000000).count)\n","user":"anonymous","dateUpdated":"2017-08-24T17:59:15+0000","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"scala","editOnDblClick":false},"editorMode":"ace/mode/scala"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"\nres241: (Long, Long) = (196630,1458)\n"}]},"apps":[],"jobName":"paragraph_1503523530787_694841550","id":"20170823-212530_423689792","dateCreated":"2017-08-23T21:25:30+0000","dateStarted":"2017-08-24T17:59:15+0000","dateFinished":"2017-08-24T17:59:26+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:7240"},{"text":"%md\n\n#### Of the companies with annual revenue > $1M and executive compensation > $1M, which companies pay out the highest amounts to executives?","user":"anonymous","dateUpdated":"2017-08-24T18:00:12+0000","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<h4>Of the companies with annual revenue > $1M and executive compensation > $1M, which companies pay out the highest amounts to executives?</h4>\n"}]},"apps":[],"jobName":"paragraph_1503521541281_-471734607","id":"20170823-205221_1236026791","dateCreated":"2017-08-23T20:52:21+0000","dateStarted":"2017-08-24T18:00:12+0000","dateFinished":"2017-08-24T18:00:12+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:7241"},{"text":"percentExecComp\n    .filter($\"annualRevenue\">=1000000)\n    .filter($\"maxExecCompensation\">=1000000)\n    .sort($\"percent\".desc)\n    .show()","user":"anonymous","dateUpdated":"2017-08-28T04:07:38+0000","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"scala","editOnDblClick":false},"editorMode":"ace/mode/scala"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"+--------------------+-------------------+-------------+------------------+\n|          vendorName|maxExecCompensation|annualRevenue|           percent|\n+--------------------+-------------------+-------------+------------------+\n|RESEARCH ANALYSIS...|           1.704E11|       2.15E7| 7925.581395348837|\n|RESEARCH ANALYSIS...|           1.704E11|        3.2E7|            5325.0|\n| SCHAFER CORPORATION|   9.53842289652E11|8.499999744E9| 112.2167433387631|\n|SCHAFER AEROSPACE...|   9.53842289652E11|        8.5E9|112.21673995905883|\n|    ASE DIRECT, INC.|         1.180767E8|    1450000.0| 81.43220689655172|\n|    A S E DIRECT LLC|         1.180767E8|    1450000.0| 81.43220689655172|\n|DIXON GROUP, INC....|              1.0E8|    6000000.0|16.666666666666668|\n|KAB LABORATORIES INC|        3.3451157E7|    2273000.0|  14.7167430708315|\n|      ACTIONET, INC.|        4.1775459E7|    2884009.0|14.485204103038514|\n| BLB RESOURCES, INC.|        2.3656825E7|    1750000.0|13.518185714285714|\n|IDENTIFICATION TE...|         1.206072E7|    1200000.0|           10.0506|\n|BRANDYWINE PHOTON...|           1.5312E7|    1689609.0| 9.062451726997192|\n|LUKE & ASSOCIATES...|        1.7008926E7|    1935649.0|  8.78719540577863|\n|PARAGON TECHNICAL...|           1.0658E8|    1.26974E7| 8.393844409091624|\n|FUTURE CARE HEALT...|       1.49900992E8|  1.8171807E7|  8.24909663634442|\n|E3 FEDERAL SOLUTI...|         1.784748E7|    2500000.0|          7.138992|\n|ACTIONET INCORPOR...|        4.1775459E7|    6400000.0|     6.52741546875|\n|IDS INTERNATIONAL...|        2.0604121E7|    3366667.0| 6.120035334649967|\n|    ASE DIRECT, INC.|         1.180767E8|        2.2E7|5.3671227272727275|\n|ATMOSPHERIC TECHN...|         1.533965E7|    2879063.0|  5.32800081137509|\n+--------------------+-------------------+-------------+------------------+\nonly showing top 20 rows\n\n"}]},"apps":[],"jobName":"paragraph_1503521425574_-977095448","id":"20170823-205025_971063320","dateCreated":"2017-08-23T20:50:25+0000","dateStarted":"2017-08-28T04:06:56+0000","dateFinished":"2017-08-28T04:06:58+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:7242"},{"text":"%md \n\n#### What companies have the highest executive pay (as a percentage of sum dollars obligated)?","user":"anonymous","dateUpdated":"2017-08-24T17:58:42+0000","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<h4>What companies have the highest executive pay (as a percentage of sum dollars obligated)?</h4>\n"}]},"apps":[],"jobName":"paragraph_1503519998403_-357082724","id":"20170823-202638_1077774614","dateCreated":"2017-08-23T20:26:38+0000","dateStarted":"2017-08-24T17:58:42+0000","dateFinished":"2017-08-24T17:58:42+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:7243"},{"text":"val joined_df = dollarsObligated.toDF.join(\r    execCompensation.toDF, \"vendorName\");\r\r \rjoined_df.show()","user":"anonymous","dateUpdated":"2017-08-24T17:59:26+0000","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"scala","editOnDblClick":false},"editorMode":"ace/mode/scala"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"+--------------------+---------------------+-------------------+\n|          vendorname|sum(dollarsobligated)|maxExecCompensation|\n+--------------------+---------------------+-------------------+\n|SCHAFER AEROSPACE...|              50000.0|   9.53842289652E11|\n| SCHAFER CORPORATION|  5.523365843000001E7|   9.53842289652E11|\n|RESEARCH ANALYSIS...|                  0.0|           1.704E11|\n|RESEARCH ANALYSIS...|           -2285908.6|           1.704E11|\n|FUTURE CARE HEALT...|    595829.1399999999|       1.49900992E8|\n|    ASE DIRECT, INC.|    6241757.649999998|         1.180767E8|\n|    A S E DIRECT LLC|   2009835.9099999997|         1.180767E8|\n|PARAGON TECHNICAL...| 1.0600045000000002E7|           1.0658E8|\n|DIXON GROUP, INC....|            -33444.09|              1.0E8|\n|         FORCE 3 INC|        1.111120801E7|        8.8277633E7|\n|       FORCE 3, INC.|  6.932773838999999E7|        8.8277633E7|\n|FORCE 3 INCORPORATED|           6477692.14|        8.8277633E7|\n|           ECCO GMBH|    5067407.069999999|        7.4968679E7|\n|ENVIRONMENTAL CHE...| 4.2892809830000006E7|        6.5707594E7|\n|     AM GENERAL, LLC|  5.318833116000001E7|        5.2454072E7|\n|ENVIRONMENTAL CHE...| 4.2892809830000006E7|        4.9559414E7|\n|ENVIRONMENTAL CHE...|             762515.8|        4.9559414E7|\n|ENVIRONMENTAL CHE...| 1.4666534420000002E7|        4.9559414E7|\n|CHARLES RIVER ANA...| 3.4977528980000004E7|         4.671253E7|\n|AXIOM RESOURCE MA...|        2.360614733E7|        4.5750944E7|\n+--------------------+---------------------+-------------------+\nonly showing top 20 rows\n\n\njoined_df: org.apache.spark.sql.DataFrame = [vendorname: string, sum(dollarsobligated): double ... 1 more field]\n"}]},"apps":[],"jobName":"paragraph_1503513981727_-1664270066","id":"20170823-184621_186689146","dateCreated":"2017-08-23T18:46:21+0000","dateStarted":"2017-08-24T17:59:26+0000","dateFinished":"2017-08-24T17:59:41+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:7244"},{"text":"%md \n\nmaxExecCompensation is often reported as 0.0 or 1.0.","user":"anonymous","dateUpdated":"2017-08-24T18:00:19+0000","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<p>maxExecCompensation is often reported as 0.0 or 1.0.</p>\n"}]},"apps":[],"jobName":"paragraph_1503520270784_-501765279","id":"20170823-203110_1131993706","dateCreated":"2017-08-23T20:31:10+0000","dateStarted":"2017-08-24T18:00:19+0000","dateFinished":"2017-08-24T18:00:19+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:7245"},{"text":"joined_df.select($\"vendorname\",\n                 $\"maxExecCompensation\", \n                 $\"sum(dollarsobligated)\", \n                 ($\"sum(dollarsobligated)\" / $\"maxExecCompensation\").as(\"percent\"))\n                 .sort($\"percent\".desc).show()","user":"anonymous","dateUpdated":"2017-08-24T17:59:27+0000","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"scala","editOnDblClick":false},"editorMode":"ace/mode/scala"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"+--------------------+-------------------+---------------------+--------------------+\n|          vendorname|maxExecCompensation|sum(dollarsobligated)|             percent|\n+--------------------+-------------------+---------------------+--------------------+\n|LAWRENCE LIVERMOR...|                1.0|      1.42804417076E9|     1.42804417076E9|\n|COUNTERTRADE PROD...|                1.0| 1.5911857584999987E8|1.5911857584999987E8|\n|GOODRICH CORPORATION|                1.0|       1.3467258139E8|      1.3467258139E8|\n|CSI AVIATION SERV...|                1.0|  9.029598071000001E7| 9.029598071000001E7|\n|WILDFLOWER INTERN...|                1.0|  5.143629600000001E7| 5.143629600000001E7|\n|SIEMENS GOVERNMEN...|                1.0|        3.423444534E7|       3.423444534E7|\n|NCR GOVERNMENT SY...|                1.0| 2.8042943869999997E7|2.8042943869999997E7|\n|SIMMONDS PRECISIO...|                1.0| 2.4675061740000002E7|2.4675061740000002E7|\n|      HUMANTOUCH LLC|                1.0| 2.1046803720000003E7|2.1046803720000003E7|\n|SUPERTEL NETWORK ...|                1.0|           5805720.44|          5805720.44|\n|PEARSON ENGINEERI...|                1.0|           5600751.34|          5600751.34|\n|TATITLEK CONSTRUC...|                1.0|    4530614.670000001|   4530614.670000001|\n|PROTECTION STRATE...|                1.0|    4276807.710000001|   4276807.710000001|\n|CARLETON TECHNOLO...|                1.0|           3964837.21|          3964837.21|\n|SYLVAIN ANALYTICS...|                1.0|           3919515.84|          3919515.84|\n|     HUMANTOUCH, LLC|                1.0|           3770041.14|          3770041.14|\n|WKF FRIEDMAN ENTR...|                1.0|            3033859.2|           3033859.2|\n|   MARY HELEN PARKEY|                1.0|   2325685.7399999998|  2325685.7399999998|\n|          SBALLIANCE|                1.0|           2099532.23|          2099532.23|\n|IAP-LEOPARDO CONS...|               10.0|          1.9258635E7|           1925863.5|\n+--------------------+-------------------+---------------------+--------------------+\nonly showing top 20 rows\n\n"}]},"apps":[],"jobName":"paragraph_1503518663798_1169513376","id":"20170823-200423_1908089949","dateCreated":"2017-08-23T20:04:23+0000","dateStarted":"2017-08-24T17:59:27+0000","dateFinished":"2017-08-24T17:59:56+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:7246"},{"text":"%md\n\n#### How many companies have reported executive compensation?","user":"anonymous","dateUpdated":"2017-08-24T17:58:42+0000","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<h4>How many companies have reported executive compensation?</h4>\n"}]},"apps":[],"jobName":"paragraph_1503520568892_948583188","id":"20170823-203608_689502656","dateCreated":"2017-08-23T20:36:08+0000","dateStarted":"2017-08-24T17:58:42+0000","dateFinished":"2017-08-24T17:58:42+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:7247"},{"text":"(joined_df.count, joined_df.filter($\"maxExecCompensation\" > 0).count)\n","user":"anonymous","dateUpdated":"2017-08-24T17:59:42+0000","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"scala","editOnDblClick":false},"editorMode":"ace/mode/scala"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"\nres245: (Long, Long) = (154596,4536)\n"}]},"apps":[],"jobName":"paragraph_1503508075663_181523723","id":"20170823-170755_533116799","dateCreated":"2017-08-23T17:07:55+0000","dateStarted":"2017-08-24T17:59:42+0000","dateFinished":"2017-08-24T18:00:29+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:7248"},{"text":"%md\n\n#### How many companies have reported executive compensation over $1M?","user":"anonymous","dateUpdated":"2017-08-24T17:58:42+0000","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<h4>How many companies have reported executive compensation over $1M?</h4>\n"}]},"apps":[],"jobName":"paragraph_1503520585698_-937825345","id":"20170823-203625_1278026986","dateCreated":"2017-08-23T20:36:25+0000","dateStarted":"2017-08-24T17:58:42+0000","dateFinished":"2017-08-24T17:58:42+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:7249"},{"text":"(joined_df.count, joined_df.filter($\"maxExecCompensation\" >= 1000000).count)\n","user":"anonymous","dateUpdated":"2017-08-24T17:59:56+0000","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"scala","editOnDblClick":false},"editorMode":"ace/mode/scala"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"\nres246: (Long, Long) = (154596,799)\n"}]},"apps":[],"jobName":"paragraph_1503520483340_136238698","id":"20170823-203443_419067401","dateCreated":"2017-08-23T20:34:43+0000","dateStarted":"2017-08-24T17:59:56+0000","dateFinished":"2017-08-24T18:00:57+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:7250"},{"text":"%md \n\n#### Of the companies that pay executives more than $1M, which have the highest executive pay (as a percentage of sum dollars obligated)?","user":"anonymous","dateUpdated":"2017-08-24T18:01:01+0000","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<h4>Of the companies that pay executives more than $1M, which have the highest executive pay (as a percentage of sum dollars obligated)?</h4>\n"}]},"apps":[],"jobName":"paragraph_1503520864105_-1357390993","id":"20170823-204104_1307679349","dateCreated":"2017-08-23T20:41:04+0000","dateStarted":"2017-08-24T18:01:01+0000","dateFinished":"2017-08-24T18:01:01+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:7251"},{"text":"joined_df.select($\"vendorname\",\n                 $\"maxExecCompensation\", \n                 $\"sum(dollarsobligated)\", \n                 ($\"sum(dollarsobligated)\" / $\"maxExecCompensation\").as(\"percent\"))\n                 .filter($\"maxExecCompensation\" >= 1000000)\n                 .sort($\"percent\".desc).show()","user":"anonymous","dateUpdated":"2017-08-24T18:00:29+0000","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"scala","editOnDblClick":false},"editorMode":"ace/mode/scala"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"+--------------------+-------------------+---------------------+------------------+\n|          vendorname|maxExecCompensation|sum(dollarsobligated)|           percent|\n+--------------------+-------------------+---------------------+------------------+\n|ELECTRIC BOAT COR...|          1334063.0|  5.024750978549999E9| 3766.502015684416|\n|ELECTRIC BOAT COR...|          1919901.0|  5.024750978549999E9|2617.1927503293136|\n|ELECTRIC BOAT COR...|          2010674.0|  5.024750978549999E9| 2499.038122813544|\n|         SANDIA CORP|          1337700.0| 2.8087110097799997E9|2099.6568810495623|\n|ELECTRIC BOAT COR...|          2533838.0|  5.024750978549999E9|1983.0592873538085|\n|LOS ALAMOS NATION...|          1082776.0| 2.0885016060100002E9| 1928.839950285193|\n|LOS ALAMOS NATION...|          1174339.0| 2.0885016060100002E9|1778.4486472900928|\n|UNITEDHEALTH MILI...|          1461163.0| 2.5053445468599997E9|1714.6235887850976|\n|HEALTH NET FEDERA...|          2157971.0|      2.78254386023E9|1289.4259747837204|\n|HEALTH NET FEDERA...|          2359163.0|      2.78254386023E9|1179.4623178771453|\n|HEALTH NET FEDERA...|          2382560.0|      2.78254386023E9|1167.8798688091802|\n|UNITED LAUNCH SER...|          1971781.0| 2.0998831267800002E9|1064.9677255131276|\n|UNITED LAUNCH SER...|          2098482.0| 2.0998831267800002E9|1000.6676858700719|\n|GENERAL ATOMICS A...|          1794950.0| 1.7455848524600003E9| 972.4977589682165|\n|UNITED LAUNCH SER...|          2254377.0| 2.0998831267800002E9|  931.469371263103|\n|GENERAL ATOMICS A...|          1951800.0| 1.7455848524600003E9| 894.3461689005022|\n|GENERAL DYNAMICS ...|          1690862.0| 1.4189483724899998E9| 839.1863868784086|\n|BATH IRON WORKS C...|          1188820.0|  9.532424104399999E8| 801.8391433858784|\n|     AUSTAL USA, LLC|          1076337.0|       8.3668829414E8|    777.3478883844|\n|GENERAL DYNAMICS ...|          1264203.0|       9.5070518142E8| 752.0193999065024|\n+--------------------+-------------------+---------------------+------------------+\nonly showing top 20 rows\n\n"}]},"apps":[],"jobName":"paragraph_1503520640526_323052640","id":"20170823-203720_1796917841","dateCreated":"2017-08-23T20:37:20+0000","dateStarted":"2017-08-24T18:00:29+0000","dateFinished":"2017-08-24T18:01:10+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:7252"},{"text":"%md\n\n#### Further Analysis\n\n* Need to deal with duplicate and differing vendor information. \n* Could some ML approach be applied to match vendors like \"BOEING COMPANY, THE\" and \"THE BOEING COMPANY\".","user":"anonymous","dateUpdated":"2017-08-24T17:58:42+0000","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<h4>Further Analysis</h4>\n<ul>\n<li>Need to deal with duplicate and differing vendor information.</li>\n<li>Could some ML approach be applied to match vendors like &ldquo;BOEING COMPANY, THE&rdquo; and &ldquo;THE BOEING COMPANY&rdquo;.</li>\n</ul>\n"}]},"apps":[],"jobName":"paragraph_1503467400355_-1573611964","id":"20170823-055000_121498953","dateCreated":"2017-08-23T05:50:00+0000","dateStarted":"2017-08-24T17:58:42+0000","dateFinished":"2017-08-24T17:58:42+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:7253"},{"text":"\n","user":"anonymous","dateUpdated":"2017-08-24T17:58:42+0000","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"scala","editOnDblClick":false},"editorMode":"ace/mode/scala"},"settings":{"params":{},"forms":{}},"apps":[],"jobName":"paragraph_1503523893612_-865100407","id":"20170823-213133_158030774","dateCreated":"2017-08-23T21:31:33+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:7254"}],"name":"Scala Spark Project","id":"2CRFTQ61R","angularObjects":{"2BRWU4WXC:shared_process":[],"2AM1YV5CU:shared_process":[],"2AJXGMUUJ:shared_process":[],"2ANGGHHMQ:shared_process":[],"2AKK3QQXU:shared_process":[]},"config":{"looknfeel":"default","personalizedMode":"false"},"info":{}}