Skip to content

Commit

Permalink
Merge pull request #6 from Fleid/Pestering
Browse files Browse the repository at this point in the history
Pestering
  • Loading branch information
Fleid authored Mar 31, 2020
2 parents 0183abb + 75ee817 commit 458f148
Show file tree
Hide file tree
Showing 55 changed files with 2,761 additions and 495 deletions.
4 changes: 4 additions & 0 deletions .vscode/settings.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,14 @@
"CICD",
"MYINPUT",
"SOLUTIONFOLDERPATH",
"SOURCESDIRECTORY",
"asaproj",
"asaql",
"aut",
"jsondiffpatch",
"prun",
"tabler",
"toolset",
"unittest"
]
}
23 changes: 0 additions & 23 deletions ASAHelloWorld/ASAHelloWorld.asaproj

This file was deleted.

107 changes: 56 additions & 51 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Unit testing Azure Stream Analytics

Unit testing for [Azure Stream Analytics](https://docs.microsoft.com/en-us/azure/stream-analytics/) (ASA), the serverless [complex-event-processing](https://en.wikipedia.org/wiki/Complex_event_processing) service running in Azure.
[PowerShell Gallery module](https://www.powershellgallery.com/packages/asa.unittest/) for unit testing [Azure Stream Analytics](https://docs.microsoft.com/en-us/azure/stream-analytics/) (ASA) jobs, the serverless [complex-event-processing](https://en.wikipedia.org/wiki/Complex_event_processing) service running in Azure.

In this article:

Expand All @@ -18,19 +18,20 @@ In this article:
- [Troubleshooting](https://github.com/Fleid/asa.unittest#Troubleshooting)
- [Internal Details](https://github.com/Fleid/asa.unittest#Internal-Details)
- [Change Log](https://github.com/Fleid/asa.unittest#Change-Log)
- [Thanks](https://github.com/Fleid/asa.unittest#Thanks)

***

## Description

### Context

At the time of writing, the major IDEs that support ASA ([VSCode](https://code.visualstudio.com/) and [Visual Studio](https://visualstudio.microsoft.com/vs/)) don't offer unit testing for it natively.
At the time of writing, the major IDEs that support ASA ([VSCode](https://code.visualstudio.com/) and [Visual Studio](https://visualstudio.microsoft.com/vs/)) do not offer native unit testing capabilities.

This solution intends to fill that gap by enabling:

- fully local, repeatable executions over multiple test cases
- automated evaluation of the resulting outputs against the expected ones
- automated evaluation of the resulting outputs against expected ones

For that it leverages the **local testing with sample data** capabilities of either [VSCode](https://docs.microsoft.com/en-us/azure/stream-analytics/visual-studio-code-local-run) or [Visual Studio](https://docs.microsoft.com/en-us/azure/stream-analytics/stream-analytics-vs-tools-local-run), as unit testing should not rely on external services (no live input).

Expand All @@ -44,7 +45,7 @@ The whole thing is wired together in a **PowerShell** script based on a predefin

*[figure 1 - High level overview](https://github.com/Fleid/fleid.github.io/blob/master/_posts/202001_asa_unittest/ut_overview.png?raw=true)*

This repository provides an **installation script**, in addition to the test script, to automate most of the setup. This installation script also allows automated executions in a continuous build pipeline such as **Azure DevOps Pipelines**.
This repository provides an **installation script** (`New-AutProject`), in addition to the test script (`Start-AutRun`), to automate most of the setup. This installation script also allows automated executions in a continuous build pipeline such as **Azure DevOps Pipelines**.

Please note that this solution is currently available **only on Windows** as it depends on *Microsoft.Azure.StreamAnalytics.CICD*.

Expand Down Expand Up @@ -76,42 +77,56 @@ From there, the installation script will take care of the other dependencies (in

To be noted that those requirements are installed by default on every Azure DevOps Pipelines agents.

### Fixture structure

The scripts will expect the following folder structure to run properly:

- **mySolutionFolder** <- *Potentially new top solution folder*
- **ASATest1** <- *Existing ASA project folder, containing the `.asaql` file and inputs folder*
- **ASATest1.Tests** <- *New folder for the test project*
- 1_arrange <- *New folder that will contain test cases*
- 2_act <- *New folder that will contain dependencies and scripts*
- 3_assert <- *New folder that will contain test run results*

The step-by-step processes below explain how to set up this environment from scratch.

### Hello World

The following steps show how to download and run the solution with the included Hello World ASA project:

1. Check all requirements are installed
1. Clone/download this repository (it includes a basic ASA project `ASAHelloWorld` and a couple of pre-configured tests in *unittest\1_arrange* )
1. **Only once** - execute the installer in the *unittest\2_act* folder: `Install-AutToolset.ps1`
- Open a **Powershell** host (terminal, ISE...)
- Navigate to `asa.unittest\unittest\2_act`
- Run `.\Install-AutToolset.ps1 -solutionPath "C:\Users\florian\Repos\asa.unittest" -verbose` with the right `-solutionPath` (absolute paths)
1. Check all the [requirements](https://github.com/Fleid/asa.unittest#Requirements) are installed
1. Import the module from the PowerShell Gallery
- Open a **Powershell** host (terminal, ISE, VSCode...)
- Run `Install-Module -Name asa.unittest` to import the module from the [PowerShell Gallery](https://www.powershellgallery.com/packages/asa.unittest)
1. Clone/download the [Examples folder](https://github.com/Fleid/asa.unittest/tree/master/examples) from this repository, save it to a convenient location (`C:\temp\examples` from now on)
1. **Only once** - execute the installer: `New-AutProject`
- In the **Powershell** host
- Run `New-AutProject -installPath "C:\temp\examples\ASAHelloWorld.Tests" -verbose` with `installPath` the absolute path to the **test folder**, **not** the ASA project folder
- ![Screenshot of a terminal run of the installation script](https://github.com/Fleid/fleid.github.io/blob/master/_posts/202001_asa_unittest/ut_install_terminal.png?raw=true)
- In case of issues see [troubleshooting](https://github.com/Fleid/asa.unittest#Troubleshooting)
1. Execute the test runner in the *unittest\2_act* folder: `Start-AutRun.ps1`
- Open a **Powershell** host (terminal, ISE...)
- Navigate to `asa.unittest\unittest\2_act`
- Run `.\Start-AutRun.ps1 -asaProjectName "ASAHelloWorld" -solutionPath "C:\Users\florian\Repos\asa.unittest" -assertPath "C:\Users\florian\Repos\asa.unittest\unittest\3_assert"-verbose` with the right `-solutionPath` and `-assertPath` (absolute paths)
1. Execute the test runner: `Start-AutRun`
- In the **Powershell** host
- Run `Start-AutRun -solutionPath "C:\temp\examples" -asaProjectName "ASAHelloWorld" -verbose` with `solutionPath` the absolute path to the folder containing both the ASA and the Test projects. `Start-AutRun` offers additional parameters that can be discovered via its help
- ![Screenshot of a terminal run of the installation script](https://github.com/Fleid/fleid.github.io/blob/master/_posts/202001_asa_unittest/ut_prun_terminal.png?raw=true)
- Here it is expected that the test ends with 2 errors, in test case *003*
- Here it is expected that the test ends with 2 errors, in test case *003*. The folder `C:\temp\examples\ASAHelloWorld.Tests\3_assert` will contain the full trace of the run
- In case of issues see [troubleshooting](https://github.com/Fleid/asa.unittest#Troubleshooting)

### Installation

The following steps show how to download and run the solution on an existing ASA project:

1. Check all requirements are installed
1. If it doesn't exist, **create a solution folder** (simple top folder)
1. Prepare the ASA Project
- If it doesn't exist, **create a solution folder** (simple top folder, `C:\temp\examples` in the HelloWorld above)
- Copy or move the existing ASA project to the solution folder
- ~~If the project was developed with VSCode (not necessary for Visual Studio), add an `.asaproj` file to the ASA project as explained below~~
- In ASA, add local inputs for every source used in the query (see [VSCode](https://docs.microsoft.com/en-us/azure/stream-analytics/visual-studio-code-local-run) / [Visual Studio](https://docs.microsoft.com/en-us/azure/stream-analytics/stream-analytics-vs-tools-local-run))
1. Clone/download this repository, copy or move the *unittest* folder to the solution folder
1. **Only once** - execute the installer in the *unittest\2_act* folder: `Install-AutToolset.ps1`
- Open a **Powershell** host (terminal, ISE...)
- Navigate to `unittest\2_act` in the solution folder
- Run `.\Install-AutToolset.ps1 -solutionPath "C:\<SOLUTIONFOLDERPATH>" -verbose` with the right `-solutionPath` (absolute paths)
1. **Only once** - Import the module from the PowerShell Gallery
- Open a **Powershell** host (terminal, ISE, VSCode...)
- Run `Install-Module -Name asa.unittest` to import the module from the [PowerShell Gallery](https://www.powershellgallery.com/packages/asa.unittest)
- In case of issues see [troubleshooting](https://github.com/Fleid/asa.unittest#Troubleshooting)
1. **Only once** - execute the installer: `New-AutProject`
- In the **Powershell** host
- Run `New-AutProject -installPath "MySolutionPath\MyTestFolder" -verbose` with `installPath` the absolute path to the **test folder**, **not** the ASA project folder. Usually the `MyTestFolder` sub-folder is of the form `MyAsaProject.Tests`

***

Expand All @@ -122,13 +137,12 @@ The following steps show how to download and run the solution on an existing ASA
Once the [installation](https://github.com/Fleid/asa.unittest#Installation) is done:

1. [Configure a test case](https://github.com/Fleid/asa.unittest#Configuring-a-test-case)
1. Execute the test runner in the *unittest\2_act* folder: `Start-AutRun.ps1`
1. Execute the test runner: `Start-AutRun`
- Open a **Powershell** host (terminal, ISE...)
- Navigate to `unittest\2_act` in the solution folder
- Run `.\Start-AutRun.ps1 -asaProjectName "<ASAPROJECTNAME>" -solutionPath "C:\<SOLUTIONFOLDERPATH>" -assertPath "C:\<SOLUTIONFOLDERPATH>\unittest\3_assert"-verbose` with the right `-solutionPath` and `-assertPath` (absolute paths)
- Run `Start-AutRun -solutionPath "MySolutionPath" -asaProjectName "MyAsaProject" -verbose` with `solutionPath` the absolute path to the folder containing both the ASA and the Test projects. `Start-AutRun` offers additional parameters that can be discovered via its help
- In case of issues see **troubleshooting**

The recommended way of running jobs is via a terminal window.
For local development, the recommended way of running jobs is via a terminal window.

### Configuring a test case

Expand Down Expand Up @@ -185,11 +199,12 @@ Note that both scripts use default values for most parameters. These default val

The mandatory parameters are:

- For the **installation script** (`Install-AutToolset`)
- `$unittestFolder` if it's not the default (`unittest`)
- For the **installation script** (`New-AutProject`)
- `$installPath`, the folder containing the test fixture
- For the **test runner** (`Start-AutRun`)
- `$asaProjectName`
- `$unittestFolder` if it's not the default (`unittest`)
- `$asaProjectName`
- `$unittestFolder` is defaulted to `$asaProjectName.Tests`
- `$solutionPath` is defaulted to `$ENV:BUILD_SOURCESDIRECTORY`

### Troubleshooting

Expand All @@ -211,19 +226,16 @@ PowerShell [remoting for jobs](https://docs.microsoft.com/en-us/powershell/modul

## Internal details

**(This needs to be updated to reflect the new parallel run workflow)**

![figure 2 - Detailed overview](https://github.com/Fleid/fleid.github.io/blob/master/_posts/202001_asa_unittest/ut_overviewFull.png?raw=true)

*[figure 2 - Detailed overview](https://github.com/Fleid/fleid.github.io/blob/master/_posts/202001_asa_unittest/ut_overviewFull.png?raw=true)*

### Change Log

- February 2020 :
- March 2020 :
- Full refactoring to allow publication in the PowerShell Gallery
- Complete unit test coverage (Pester) and linting
- February 2020 :
- Automated the generation of the XML asaproj file (required by sa.exe) from the JSON one (for VSCode project)
- Renamed scripts to follow PowerShell standards, with backward compatibility

### Scenario and components
### Components

This solution uses the following components:

Expand All @@ -233,21 +245,14 @@ This solution uses the following components:
- the [npm CLI](https://docs.npmjs.com/cli-documentation/) to install the package above, available with [Node.JS](https://nodejs.org/en/download/)
- [PowerShell](https://github.com/PowerShell/PowerShell/releases) as the shell to run intermediary tasks and execute the required commands

These components are used in a script as follow:

![figure 1 - Schema of the unit testing setup - it is detailed below](https://github.com/Fleid/fleid.github.io/blob/master/_posts/202001_asa_unittest/ut_solution.png?raw=true)

The script will expect the following folder structure to run properly:

- **mySolutionFolder** <- *Potentially new top solution folder*
- **ASATest1** <- *Existing ASA project folder, containing the `.asaql` file and inputs folder*
- **unittest** <- *New folder for the test project*
- 1_arrange <- *New folder that will contain test cases*
- 2_act <- *New folder that will contain dependencies and scripts*
- 3_assert <- *New folder that will contain test run results*

### Shortcomings

- Slow execution
- No integration to the usual IDEs
- No cross-platform options

### Thanks

- [Kevin Marquette](https://twitter.com/KevinMarquette) for his [invaluable](https://powershellexplained.com/2017-01-21-powershell-module-continious-delivery-pipeline/) blog on PowerShell
- [jsondiffpatch](https://github.com/benjamine/JsonDiffPatch) : Diff & patch JavaScript objects
- [Tabler icons](https://github.com/tabler/tabler-icons) : A set of over 400 free MIT-licensed high-quality SVG icons for you to use in your web projects
98 changes: 98 additions & 0 deletions asa.unittest.tests/Get-AutFieldFromFileInfo.Tests.ps1
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
### If tests are in ModuleRoot\Whatever, and scripts reachable via a module at ModuleRoot\Module\Module.psm1

$projectRoot = Resolve-Path "$PSScriptRoot\.." #ModuleRoot\Whatever becomes \ModuleRoot with ..
$moduleRoot = Split-Path (Resolve-Path "$projectRoot\*\*.psd1") #Look for the manifest to get to ModuleRoot\Module
$moduleName = Split-Path $moduleRoot -Leaf #Extract the module name from above
Import-Module (Join-Path $moduleRoot "$moduleName.psm1") -force

#############################################################################################################
# Invoke-Pester .\Get-AutFieldFromFileInfo.Tests.ps1 -CodeCoverage .\..\asa.unittest\private\Get-AutFieldFromFileInfo.ps1


Describe "Get-AutFieldFromFileInfo parameters" {
InModuleScope $moduleName {
function GetFullPath {
Param(
[string] $Path
)
return $Path.Replace('TestDrive:', (Get-PSDrive TestDrive).Root)
}


$t_testPath = "TestDrive:\users\fleide\Repos\asa.unittest\examples\ASAHelloWorld.Tests\1_arrange"
New-Item -Path $t_testPath -ItemType Directory
New-item -Path $t_testPath -ItemType File -Name "001~Input~hwsource~nominal.csv"
New-item -Path $t_testPath -ItemType File -Name "001~Output~outputall.json"

$t_thisFileInfo = (Get-ChildItem -Path $t_testPath -File)
$t_separatorOfFields = "~"
$t_numberOfFields = 4

$Output_1 = New-Object PSObject -Property @{
FullName="001~Input~hwsource~nominal.csv"
FilePath= GetFullPath("TestDrive:\Users\fleide\Repos\asa.unittest\examples\ASAHelloWorld.Tests\1_arrange\001~Input~hwsource~nominal.csv")
Basename= "001~Input~hwsource~nominal"
basename0="001"
basename1="Input"
basename2="hwsource"
basename3="nominal"
}

$Output_2 = New-Object PSObject -Property @{
FullName="001~Output~outputall.json"
FilePath= GetFullPath("TestDrive:\Users\fleide\Repos\asa.unittest\examples\ASAHelloWorld.Tests\1_arrange\001~Output~outputall.json")
Basename= "001~Output~outputall"
basename0="001"
basename1="Output"
basename2="outputall"
}

#Mock Test-Path {return $true}

It "runs with a valid set of parameters (output1)" {
$actual = Get-AutFieldFromFileInfo `
-t $t_thisFileInfo[0] `
-s $t_separatorOfFields `
-n $t_numberOfFields

$t = $true

foreach($actual_properties in $actual.PSObject.Properties)
{
$t = $t -and ($Output_1.($actual_properties.Name) -eq $actual_properties.value)
}

$t | Should -be $true
}

$t_numberOfFields = 3
It "runs with a valid set of parameters (output2)" {
$actual = Get-AutFieldFromFileInfo `
-t $t_thisFileInfo[1] `
-s $t_separatorOfFields `
-n $t_numberOfFields

$t = $true

foreach($actual_properties in $actual.PSObject.Properties)
{
$t = $t -and ($Output_2.($actual_properties.Name) -eq $actual_properties.value)
}

$t | Should -be $true
}

It "runs from the pipeline" {
$actual = $t_thisFileInfo[0] | Get-AutFieldFromFileInfo -s "~" -n 4

$t = $true

foreach($actual_properties in $actual.PSObject.Properties)
{
$t = $t -and ($Output_1.($actual_properties.Name) -eq $actual_properties.value)
}

$t | Should -be $true
}
}
}
Loading

0 comments on commit 458f148

Please sign in to comment.