730 update docker resources (#815)

* Create copy of rita config file and replace localhost with db

* Fix typo in Docker Usage guide

* Update Docker Usage guide to assume user is not part of docker group

Dockerfile

@@ -24,7 +24,7 @@ RUN make CGO_ENABLED=$CGO_ENABLED GOARCH=$GOARCH GOOS=$GOOS
 FROM scratch
 
 WORKDIR /
-COPY --from=rita-builder /go/src/github.com/activecm/rita/etc/rita.yaml /etc/rita/config.yaml
+COPY --from=rita-builder /go/src/github.com/activecm/rita/etc/rita_docker.yaml /etc/rita/config.yaml
 COPY --from=rita-builder /go/src/github.com/activecm/rita/rita /rita
 
 ENTRYPOINT ["/rita"]

docker-compose.yml

@@ -9,9 +9,8 @@ services:
   rita:
     image: quay.io/activecm/rita:${VERSION:-latest}
     build: .
-    links:
-      # give db an alias of "localhost" so that RITA's default config works unchanged
-      - db:localhost
+    depends_on:
+      - db
     volumes:
       - ${CONFIG:?You must provide a path to your CONFIG}:/etc/rita/config.yaml:ro
       - ${LOGS:?You must provide a path to your LOGS}:/logs:ro

docs/Docker Usage.md

@@ -10,13 +10,13 @@ You can run RITA using Docker! You have several options depending on your specif
 The easiest way is to pull down the pre-built image.
 
 ```
-docker pull quay.io/activecm/rita
+sudo docker pull quay.io/activecm/rita
 ```
 
 You can also build the image from source.
 
 ```
-docker build -t quay.io/activecm/rita .
+sudo docker build -t quay.io/activecm/rita .
 ```
 
 ## Running RITA with Docker Compose
@@ -25,24 +25,24 @@ You will need a config file where you have [put in your `InternalSubnets`](../Re
 You will also need the path to your Zeek log files.
 
 ```
-export CONIFG=/path/to/your/rita/config.yaml
+export CONFIG=/path/to/your/rita/config.yaml
 export LOGS=/path/to/your/zeek/logs
-docker-compose run --rm rita import /logs your-dataset
+sudo -E docker compose run --rm rita import /logs your-dataset
 ```
 
 Note: If you'd like to use a specific version of RITA than the default `latest` you can do so using the `VERSION` variable.
 
 ```
 export VERSION=v4.3.0
-docker-compose run --rm rita --version
+sudo -E docker compose run --rm rita --version
 ```
 
 ## Running RITA with Docker Using External Mongo
 
 If you don't need/want the convenience of Docker Compose running the Mongo server for you, you can also use RITA without it. You will need to modify RITA's config file to point to your external Mongo server and invoke RITA like this:
 
 ```
-docker run -it --rm \
+sudo docker run -it --rm \
 	-v /path/to/your/zeek/logs:/logs:ro \
 	-v /path/to/your/rita/config.yaml:/etc/rita/config.yaml:ro \
 	quay.io/activecm/rita import /logs your-dataset
@@ -53,9 +53,9 @@ docker run -it --rm \
 You can use Docker to build a statically linked RITA binary for you. This binary should be portable between Linux 64-bit systems. Once you've obtained the RITA docker image (see the "Obtaining the RITA Docker Image" section above) you can run the following commands to copy the binary to your host system.
 
 ```
-docker create --name rita quay.io/activecm/rita
-docker cp rita:/rita ./rita
-docker rm rita
+sudo docker create --name rita quay.io/activecm/rita
+sudo docker cp rita:/rita ./rita
+sudo docker rm rita
 ```
 
 Note that you will have to manually install the `config.yaml` files into `/etc/rita/` as well as create any directories referenced inside the `config.yaml` file.

etc/rita_docker.yaml

@@ -0,0 +1,291 @@
+# This section configures the connection to the MongoDB server and the database name to use
+MongoDB:
+  # See https://docs.mongodb.com/manual/reference/connection-string/
+  ConnectionString: mongodb://db:27017
+  # Example with authentication. Be sure to change the AuthenticationMechanism as well.
+  # ConnectionString: mongodb://username:password@localhost:27017
+
+  # Accepted Values: null, "SCRAM-SHA-1", "MONGODB-CR", "PLAIN"
+  # Since Mongo version 3.0 the default authentication mechanism is SCRAM-SHA-1
+  AuthenticationMechanism: null
+
+  # The time in hours before RITA's connection to MongoDB times out. 0 waits indefinitely.
+  SocketTimeout: 2
+
+  # For encrypting data on the wire between RITA and MongoDB
+  TLS:
+    Enable: false
+    #If set, RITA will verify the MongoDB certificate's hostname and validity
+    VerifyCertificate: false
+    #If set, RITA will use the provided CA file instead of the system's CA's
+    CAFile: null
+
+  # This database holds information about the procesed files and databases.
+  MetaDB: MetaDatabase
+
+Rolling:
+  # This is the default number of chunks to keep in rolling databases.
+  # This only is used if the --numchunks command argument isn't supplied.
+  DefaultChunks: 24
+
+LogConfig:
+  # LogLevel
+  # 3 = debug
+  # 2 = info
+  # 1 = warn
+  # 0 = error
+  LogLevel: 2
+
+  # LogPath is the path for Rita's logs. Make sure permissions are set accordingly.
+  # Logs will only be written here if LogToFile is true
+  RitaLogPath: /var/lib/rita/logs
+
+  LogToFile: true
+  LogToDB: true
+
+UserConfig:
+  # Number of days before checking for a new version of RITA.
+  # A value of zero here will disable checking.
+  UpdateCheckFrequency: 14
+
+Filtering:
+  # These are filters that affect the import of connection logs. They
+  # currently do not apply to dns or http logs.
+  # A good reference for networks you may wish to consider is RFC 5735.
+  # https://tools.ietf.org/html/rfc5735#section-4
+
+  # Example: AlwaysInclude: ["192.168.1.2/32"]
+  # This functionality overrides the NeverInclude and InternalSubnets
+  # section, making sure that any connection records containing addresses from
+  # this range are kept and not filtered
+  AlwaysInclude: []
+
+  # Example: NeverInclude: ["255.255.255.255/32"]
+  # This functions as a whitelisting setting, and connections involving
+  # ranges entered into this section are filtered out at import time
+  NeverInclude:
+    - 0.0.0.0/32 # "This" Host           RFC 1122, Section 3.2.1.3
+    - 127.0.0.0/8 # Loopback              RFC 1122, Section 3.2.1.3
+    - 169.254.0.0/16 # Link Local            RFC 3927
+    - 224.0.0.0/4 # Multicast             RFC 3171
+    - 255.255.255.255/32 # Limited Broadcast     RFC 919, Section 7
+    - ::1/128 # Loopback              RFC 4291, Section 2.5.3
+    - fe80::/10 # Link local            RFC 4291, Section 2.5.6
+    - ff00::/8 # Multicast             RFC 4291, Section 2.7
+
+  # Example: InternalSubnets: ["10.0.0.0/8","172.16.0.0/12","192.168.0.0/16"]
+  # This allows a user to identify their internal network, which will result
+  # in any internal to internal and external to external connections being
+  # filtered out at import time. Reasonable defaults are provided below
+  # but need to be manually verified against each installation before enabling.
+  InternalSubnets:
+    - 10.0.0.0/8 # Private-Use Networks  RFC 1918
+    - 172.16.0.0/12 # Private-Use Networks  RFC 1918
+    - 192.168.0.0/16 # Private-Use Networks  RFC 1918
+
+  # Example: AlwaysIncludeDomain: ["mydomain.com","*.mydomain.com"]
+  # This functionality overrides the NeverIncludeDomain
+  # section, making sure that any connection records containing domains
+  # that match this list are kept and not filtered
+  # NOTE: When using wildcards, make sure the added entry is in quotes,
+  #       ie, '*.mydomain.com'. Only subdomain wildcarding
+  #       (asterisk as the prefix) is supported
+  AlwaysIncludeDomain: []
+
+  # Example: NeverIncludeDomain: ["mydomain.com","*.mydomain.com"]
+  # This functions as a whitelisting setting, and connections involving
+  # ranges entered into this section are filtered out at import time
+  # NOTE: When using wildcards, make sure the added entry is in quotes,
+  #       ie, '*.mydomain.com'. Only subdomain wildcarding
+  #       (asterisk as the prefix) is supported
+  NeverIncludeDomain: []
+
+  # FilterExternalToInternal will ignore any entries where communication
+  # is occurring from an external host to an internal host
+  FilterExternalToInternal: true
+
+BlackListed:
+  Enabled: true
+  # These are blacklists built into rita-blacklist. Set these to false
+  # to disable checks against them.
+  feodotracker.abuse.ch: true
+
+  # This is the name of the database which will be created as a master list of
+  # blacklisted ips and hostnames by rita-blacklist
+  BlacklistDatabase: "rita-bl"
+
+  # These are custom blacklists that you may define. They are lists of either
+  # file paths or urls. These custom blacklists are expected to be simple,
+  # line separated text documents containing a list of blacklisted entries.
+
+  # Example: CustomIPBlacklists: ["$HOME/.rita/myIPBlacklist.txt"]
+  # myIPBlacklist.txt would look like this:
+  # 192.168.0.1
+  # 10.10.174.1
+
+  # Lists containing both IPv4 and IPv6 addresses are acceptable
+  CustomIPBlacklists: []
+  # Lists containing hostnames, domain names, and FQDNs are acceptable
+  CustomHostnameBlacklists: []
+
+Beacon:
+  Enabled: true
+  # The default minimum number of connections used for beacons analysis.
+  # Any two hosts connecting fewer than this number will not be analyzed. You can 
+  # safely increase this value to improve performance if you are not concerned
+  # about slow beacons.
+
+  # Note: Since analyzing hosts that have fewer than at least one connection per 
+  # hour could significantly increase both the analysis time and the number
+  # of false positives, 23 is the minimum allowed value for this field.
+  DefaultConnectionThresh: 23
+
+  # The score is currently comprised of a weighted average of 4 subscores.
+  # While we recommend the default setting of 0.25 for each weight, 
+  # these weights can be altered here according to your needs. 
+  # The sum of all the floating point weights must be equal to 1.
+  TimestampScoreWeight: 0.25
+  DatasizeScoreWeight: 0.25
+  DurationScoreWeight: 0.25
+  HistogramScoreWeight: 0.25
+
+  # The number of hours seen in a connection graph representation of a beacon must
+  # be greater than this threshold for an overall duration score to be calculated.
+  # Default value: 6
+  DurationMinHoursSeen: 6
+  # This is the minimum number of hours seen in a connection graph representation
+  # of a beacon for the consistency subscore of duration to score at 100%
+  # Default value: 12 (half the day)
+  DurationConsistencyIdealHoursSeen: 12
+
+  # The histogram score has a subscore that attempts to detect multiple 
+  # flat sections in a connection graph representation of a beacon. The 
+  # variable below controls the bucket size for grouping connections. This
+  # is expressed as a percentage of the largest connection count. For example, 
+  # if the max connection count is 400 and this variable is set to 0.05 (5%), 
+  # the bucket size will be 20 (400*0.05=20). As you make this variable 
+  # larger, the algorithm becomes more forgiving to variation. 
+  # Default value 0.05
+  HistogramBimodalBucketSize: 0.05
+  # This is the number of buckets that can be considered outliers and dropped
+  # from the calculation.
+  # Default value: 1
+  HistogramBimodalOutlierRemoval: 1
+  # This is the minimum number of hours seen in a connection graph representation
+  # of a beacon before the bimodal subscore score is used.
+  # Default value: 11 (sets the minimum coverage to just below half of the day)
+  HistogramBimodalMinHoursSeen: 11
+
+BeaconSNI:
+  Enabled: true
+  # The default minimum number of connections used for beacons SNI analysis.
+  # Any two hosts connecting fewer than this number will not be analyzed. You can 
+  # safely increase this value to improve performance if you are not concerned
+  # about slow beacons.
+
+  # Note: Since analyzing hosts that have fewer than at least one connection per 
+  # hour could significantly increase both the analysis time and the number
+  # of false positives, 23 is the minimum allowed value for this field.
+  DefaultConnectionThresh: 23
+
+  # The score is currently comprised of a weighted average of 4 subscores.
+  # While we recommend the default setting of 0.25 for each weight, 
+  # these weights can be altered here according to your needs. 
+  # The sum of all the floating point weights must be equal to 1.
+  TimestampScoreWeight: 0.25
+  DatasizeScoreWeight: 0.25
+  DurationScoreWeight: 0.25
+  HistogramScoreWeight: 0.25
+
+  # The number of hours seen in a connection graph representation of a beacon must
+  # be greater than this threshold for an overall duration score to be calculated.
+  # Default value: 6
+  DurationMinHoursSeen: 6
+  # This is the minimum number of hours seen in a connection graph representation
+  # of a beacon for the consistency subscore of duration to score at 100%
+  # Default value: 12 (half the day)
+  DurationConsistencyIdealHoursSeen: 12
+
+  # The histogram score has a subscore that attempts to detect multiple 
+  # flat sections in a connection graph representation of a beacon. The 
+  # variable below controls the bucket size for grouping connections. This
+  # is expressed as a percentage of the largest connection count. For example, 
+  # if the max connection count is 400 and this variable is set to 0.05 (5%), 
+  # the bucket size will be 20 (400*0.05=20). As you make this variable 
+  # larger, the algorithm becomes more forgiving to variation. 
+  # Default value 0.05
+  HistogramBimodalBucketSize: 0.05
+  # This is the number of buckets that can be considered outliers and dropped
+  # from the calculation.
+  # Default value: 1
+  HistogramBimodalOutlierRemoval: 1
+  # This is the minimum number of hours seen in a connection graph representation
+  # of a beacon before the subscore score is used.
+  # Default value: 11 (sets the minimum coverage to just below half of the day)
+  HistogramBimodalMinHoursSeen: 11
+
+BeaconProxy:
+  Enabled: true
+  # The default minimum number of connections used for beacons proxy analysis.
+  # Any two hosts connecting fewer than this number will not be analyzed. You can 
+  # safely increase this value to improve performance if you are not concerned
+  # about slow beacons.
+
+  # Note: Since analyzing hosts that have fewer than at least one connection per 
+  # hour could significantly increase both the analysis time and the number
+  # of false positives, 23 is the minimum allowed value for this field.
+  DefaultConnectionThresh: 23
+
+  # The score is currently comprised of a weighted average of 3 subscores.
+  # While we recommend the default setting of 0.333 for each weight, 
+  # these weights can be altered here according to your needs. 
+  # The sum of all the floating point weights must be equal to 1
+  TimestampScoreWeight: 0.333
+  DurationScoreWeight: 0.333
+  HistogramScoreWeight: 0.333
+
+  # The number of hours seen in a connection graph representation of a beacon must
+  # be greater than this threshold for an overall duration score to be calculated.
+  # Default value: 6
+  DurationMinHoursSeen: 6
+  # This is the minimum number of hours seen in a connection graph representation
+  # of a beacon for the consistency subscore of duration to score at 100%
+  # Default value: 12 (half the day)
+  DurationConsistencyIdealHoursSeen: 12
+
+  # The histogram score has a subscore that attempts to detect multiple 
+  # flat sections in a connection graph representation of a beacon. The 
+  # variable below controls the bucket size for grouping connections. This
+  # is expressed as a percentage of the largest connection count. For example, 
+  # if the max connection count is 400 and this variable is set to 0.05 (5%), 
+  # the bucket size will be 20 (400*0.05=20). As you make this variable 
+  # larger, the algorithm becomes more forgiving to variation. 
+  # Default value 0.05
+  HistogramBimodalBucketSize: 0.05
+  # This is the number of buckets that can be considered outliers and dropped
+  # from the calculation.
+  # Default value: 1
+  HistogramBimodalOutlierRemoval: 1
+  # This is the minimum number of hours seen in a connection graph representation
+  # of a beacon before the subscore score is used.
+  # Default value: 11 (sets the minimum coverage to just below half of the day)
+  HistogramBimodalMinHoursSeen: 11
+
+DNS:
+  Enabled: true
+
+UserAgent:
+  Enabled: true
+
+Strobe:
+  # This sets the maximum number of connections between any two given hosts that are stored.
+  # Connections above this limit will be deleted and not used in other analysis modules. This will
+  # also trigger an entry in the strobe module. A lower value will reduce import & analysis time and
+  # hide more potential false positives from other modules. A higher value will increase import &
+  # analysis time, increase false positives, but reduce the risk of false negatives.
+
+  # Note: Since this value must be low enough to ensure that documents that store
+  # arrays based off of connections will not grow beyond 16MB in size, the maximum value allowed
+  # for this field is:
+  #    86400 - One connection every second for 24 hours
+  ConnectionLimit: 86400