Skip to content

Scanner details

Jump to bottom
justinbastress edited this page Feb 13, 2018 · 5 revisions

Layout

Most scanners share a good deal of boilerplate, and there are some helper functions that encapsulate / standardize common operations. The integration-tests/new.sh script will set up a template for a new module, but it follows the guidelines set out here.

To make the godocs useful, modules should each be in their own package under the modules package, and their documentation should share a common format.

So, the scanner would go into zgrab2/modules/[your module]/scanner.go.

Package docs

The package docs (before the package myprotocol line) should follow roughly the following script:

// Package myprotocol provides the zgrab2 scanner module for myprotocol.
// Default Port: <your default port> (TCP)
// 
// The --some-flag changes the scan/output in this way
// The --other-flag changes the scan/output in this way
// ...<more flag descriptions>...
// Supports the standard TLS flags <unless it doesn't>
//
// The scan does X, Y and Z, unless P then it does Q.
// ...<more scan description>...
//
// The output is (description of the output).
// ...<more output description>...
package myprotocol

Types

Flags

All Flags implementations must include zgrab2.BaseFlags.

If the protocol supports TLS, it should also include zgrab2.TLSFlags unless it has very specialized requirements.

If the protocol uses UDP, it should include zgrab2.UDPFlags.

All module-specific flags should be tagged with long and description, and possibly short and default. long names should be in lisp-case.

TODO conventions on different value types (constants, bit-fields, lists and other structured data, ...)

TODO: We have discussed adding the flags to the output; to be ready for that, it may make sense to include json tags.

Godocs

The fields of the Flags struct should be individually documented, and their docs should include roughly the same content as the description tag -- e.g. allowed values and the interpretations of those values.

Scanner

A Scanner instance is created by the framework for each scan, while that instance is shared for each target. So if you need to store state for scanning an individual target, storing it in the Scanner instance will not work.

The Scan Method

The bulk of the work will go into the Scan method, which receives a ScanTarget telling it what machine to scan. The ScanTarget's IP will always be set; the Domain may or may not be present (TODO: Confirm this sentence).

Scan is invoked by the framework on each target on several threads in parallel. The Scanner is completely responsible for connecting and disconnecting from the target; there is nothing preventing it from connecting to different ports or different hosts (as can happen with e.g. HTTP redirects), or connecting to the target multiple times.

Connecting

For most purposes, the ScanTarget.Open(*BaseFlags) (or ScanTarget.OpenUDP(*BaseFlags, *UDPFlags)) will suffice -- it handles opening the connection to the appropriate host/port and setting up the connect/read/write timeouts based on the input flags (and in the UDP case, the source address/port). On success, the return value is a wrapped net.Conn object that times out on Read/Write and a nil error; on failure, the Dial error is returned. It is good practice to add a defer conn.Close() to ensure that the connection is closed when the scan is finished, even if there was a panic in the meantime.

TLS

If the scan needs to do a TLS handshake, in most cases it should use the TLSFlags.GetTLSConnection(net.Conn) method rather than directly calling tls.Client(net.Conn, tls.Config); this returns a zgrab2.TLSConnection object, which extends the standard tls.Conn object and provides its own Handshake method to do any additional checks specified by the command line flags (e.g. heartbleed).

In cases where this is not possible (for instance, using some third-party libraries), TLSFlags provides other primitives (e.g. TLSFlags.GetTLSConfig()).

The TLS log is available via TLSConnection.GetLog(), which includes both the normal zcrypto TLS log and logs for any additional scans (e.g. heartbleed). This can be called early to get a pointer that will be populated as the scan progresses, so that in the case of a failure partial logs can still be captured. However, its member fields may be overwritten, so only the outer structure should be kept.

Return values

The Scan method returns three values: the status, the result, and an optional error object.

The status gives the disposition of the scan -- if it ended successfully, it is a SCAN_SUCCESS, if it ended with a timeout, it is a SCAN_IO_TIMEOUT, etc:

TODO: Change to proper go constant names, e.g. SCAN_SUCCESS -> ScanSuccess

Status JSON value Description
SCAN_SUCCESS "success" The scan completed as expected.
SCAN_CONNECTION_TIMEOUT "connection-timeout" Could not open a connection to the target within the specified time.
SCAN_CONNECTION_REFUSED "connection-refused" The target actively refused the connection.
SCAN_CONNECTION_CLOSED "connection-closed" The target unexpectedly closed the connection (TODO: Merge with SCAN_IO_ERROR?).
SCAN_IO_ERROR "io-error" TODO: Needs to be added. Encompasses any primitive errors from conn.Read() / conn.Write()
SCAN_IO_TIMEOUT "io-timeout" The timeout condition was hit on a connection.
SCAN_PROTOCOL_ERROR "protocol-error" Received data that could not be interpreted within the protocol.
SCAN_APPLICATION_ERROR "application-error" Successfully detected the protocol, but it is reporting an error (e.g. MySQL rejecting a connection because no users are allowed from the scanner's IP address)
SCAN_UNKNOWN_ERROR "unknown-error" Something unexpected occurred and the cause could not be identified.

The result gives any information that could be gleaned from the scan -- if result is nil, the inference is that the service was not detected. So, it is possible to have a non-SCAN_SUCCESS status and still have a non-nil result.

The result is defined to be an interface{}, but there are some requirements on it -- see the ScanResult section below.

The error is an optional piece of information that gives additional context for the reason the status was not SCAN_SUCCESS.

Godocs

The Scan method should describe the scanning process in detail, step-by-step, making note of when the service is considered to be detected (i.e. when a non-nil result is returned) and where the output comes from. For example:

// Scan probes for the presence of XYZ on the target.
//  1. Connect to the configured TCP port on target (default XYZ)
//  2. Send packet A, receive response
//    - On failure, fail detection with SCAN_PROTOCOL_ERROR
//  3. Copy field X from response into field X of results
//  4. Send packet B, receive response
//   - On failure, return SCAN_PROTOCOL_ERROR with the partial results
// ...
// 10. Copy field Z from response into field Z of results
// 11. Return SCAN_SUCCESS with the results.
func (scanner *Scanner) Scan(target ScanTarget) (ScanStatus, interface{} error) {

ScanResult

The ScanResult type encapsulates the output of the scan. Its fields should all be exported, so that they are accessible by the JSON library, and they should all have the appropriate json tag.

There should be a simple mapping between the Go field name and the JSON field name -- the MixedCaps name of the Go field should be snake_case in JSON -- e.g.

type ScanResults struct {
  // MyFieldID is a string that can safely be omitted if it is empty.
  MyFieldID string   `json:"my_field_id,omitempty"`

  // MyBoolean is a boolean value, and we want to include it in the response whether the value is true or false.
  // This is a debug value that should only be returned when verbose output is enabled.
  MyBoolean bool     `json:"my_boolean" zgrab:"debug"`

  // MyOptionalInt is an optional int, where the value 0 is meaningful and distinct from "not present".
  MyOptionalInt *int `json:"my_optional_int,omitempty"`
}

By convention, unless the "nil value" of a field must be represented (e.g. an integer field where 0 is meaningful or a boolean), all fields should be omitempty. In cases where the nil value and "field not present" must be distinguished, an omitempty pointer may be used (TODO see if there is a conventional way to do this).

TODO FIXME - omitting debug fields is currently not implemented. Results can also optionally have a zgrab tag, which currently has one value defined: "debug". If present (as in the MyBoolean field in the example above), this indicates that the field should only be included in the JSON output if the caller requested verbose output.

The ScanResult's JSON encoding must match its Schema details, and must have a database- and search-friendly form:

  • No maps with unconstrained keys
  • No loops
  • No context-dependent structure
    • Simple cases can be encoded as ASN.1 CHOICE-style structures, where you define a structure with keys equal to every possibility, but only one is ever set at a time. See again for example extensions in zcrypto.py.
  • Sets should be encoded as a map[setType]bool.
  • Bit-string flags should be encoded as a set of strings (map[string]bool), where the keys are human-readable mnemonics for the individual bits (e.g. matching #defines from third-party docs).
  • Raw, unparsed output should be prefixed with Raw (e.g. RawCommandOutput []byte `json:"raw_command_output,omitempty"` )
  • Ideally it should be possible to decode the JSON output back into the original Go ScanResult instance.
  • If the protocol supports TLS, there should be a tls field at the root of the object containing the TLSConnection.GetLogs() value.

Unit testing

TODO: This needs work. The integration tests can cover the end-to-end behavior, and protocol primitives should provide their own tests (e.g. encoding / decoding packets).

Clone this wiki locally