Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Customize amount of Netflow aggregator history kept #6085

Open
wants to merge 8 commits into
base: master
Choose a base branch
from
Open

Customize amount of Netflow aggregator history kept #6085

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
bb78fb8
Add Netflow history to model
n-thumann Oct 10, 2022
88ed16d
Add Netflow history to controller
n-thumann Oct 10, 2022
16f8958
Add Netflow history template
n-thumann Oct 10, 2022
a1b1e5c
Remove history_per_resolution method
n-thumann Oct 10, 2022
9a83677
Enable loading of config from file
n-thumann Oct 10, 2022
c77279f
Add Netflow history defaults to config
n-thumann Oct 10, 2022
2bd9aee
Apply Netflow history config on cleanup
n-thumann Oct 10, 2022
61afc48
Adjust get_top_usage.py to Netflow history config
n-thumann Oct 13, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions plist
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1007,6 +1007,7 @@
/usr/local/opnsense/service/templates/OPNsense/Netflow/flowd.conf
/usr/local/opnsense/service/templates/OPNsense/Netflow/flowd.rc.conf.d
/usr/local/opnsense/service/templates/OPNsense/Netflow/flowd_aggregate.rc.conf.d
/usr/local/opnsense/service/templates/OPNsense/Netflow/flowd_aggregate.yaml
/usr/local/opnsense/service/templates/OPNsense/Netflow/netflow.conf
/usr/local/opnsense/service/templates/OPNsense/Netflow/rc.conf.d
/usr/local/opnsense/service/templates/OPNsense/Proxy/+TARGETS
Expand Down
77 changes: 77 additions & 0 deletions src/opnsense/mvc/app/controllers/OPNsense/Diagnostics/forms/netflow_capture.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,4 +48,81 @@
<advanced>true</advanced>
<help><![CDATA[Expire idle flows.]]></help>
</field>
<field>
<id>netflow.collect.history.FlowInterfaceTotals._30</id>
<label>Interface Totals History (30 sec.)</label>
<type>text</type>
<advanced>true</advanced>
<help><![CDATA[Seconds of history to keep for the 30 sec. resolution. Default is 86400 sec. (24 hours).]]></help>
</field>
<field>
<id>netflow.collect.history.FlowInterfaceTotals._300</id>
<label>Interface Totals History (5 min.)</label>
<type>text</type>
<advanced>true</advanced>
<help><![CDATA[Seconds of history to keep for the 300 sec. resolution. Default is 604800 sec. (7 days).]]></help>
</field>
<field>
<id>netflow.collect.history.FlowInterfaceTotals._3600</id>
<label>Interface Totals History (1 hour)</label>
<type>text</type>
<advanced>true</advanced>
<help><![CDATA[Seconds of history to keep for 3600 sec. resolution. Default is 2678400 sec. (31 days).]]></help>
</field>
<field>
<id>netflow.collect.history.FlowInterfaceTotals._86400</id>
<label>Interface Totals History (24 hours)</label>
<type>text</type>
<advanced>true</advanced>
<help><![CDATA[Seconds of history to keep for the 86400 sec. resolution. Default is 31536000 sec. (365 days).]]></help>
</field>
<field>
<id>netflow.collect.history.FlowDstPortTotals._300</id>
<label>Destinations Ports Totals History (5 min.)</label>
<type>text</type>
<advanced>true</advanced>
<help><![CDATA[Seconds of history to keep for the 300 sec. resolution. Default is 3600 sec. (1 hour).]]></help>
</field>
<field>
<id>netflow.collect.history.FlowDstPortTotals._3600</id>
<label>Destinations Ports Totals History (1 hour)</label>
<type>text</type>
<advanced>true</advanced>
<help><![CDATA[Seconds of history to keep for the 3600 sec. resolution. Default is 86400 sec. (24 hours).]]></help>
</field>
<field>
<id>netflow.collect.history.FlowDstPortTotals._86400</id>
<label>Destinations Ports Totals History (24 hours)</label>
<type>text</type>
<advanced>true</advanced>
<help><![CDATA[Seconds of history to keep for the 86400 sec. resolution. Default is 31536000 sec. (365 days).]]></help>
</field>
<field>
<id>netflow.collect.history.FlowSourceAddrTotals._300</id>
<label>Source Address Totals History (5 min.)</label>
<type>text</type>
<advanced>true</advanced>
<help><![CDATA[Seconds of history to keep for the 300 sec. resolution. Default is 3600 sec. (1 hour).]]></help>
</field>
<field>
<id>netflow.collect.history.FlowSourceAddrTotals._3600</id>
<label>Source Address Totals History (1 hour)</label>
<type>text</type>
<advanced>true</advanced>
<help><![CDATA[Seconds of history to keep for the 3600 sec. resolution. Default is 86400 sec. (24 hours).]]></help>
</field>
<field>
<id>netflow.collect.history.FlowSourceAddrTotals._86400</id>
<label>Source Address Totals History (24 hours)</label>
<type>text</type>
<advanced>true</advanced>
<help><![CDATA[Seconds of history to keep for the 86400 sec. resolution. Default is 31536000 sec. (365 days).]]></help>
</field>
<field>
<id>netflow.collect.history.FlowSourceAddrDetails._86400</id>
<label>Source Address Details History (24 hours)</label>
<type>text</type>
<advanced>true</advanced>
<help><![CDATA[Seconds of history to keep for the 86400 sec. resolution. Default is 5356800 sec. (62 days).]]></help>
</field>
</form>
76 changes: 76 additions & 0 deletions src/opnsense/mvc/app/models/OPNsense/Diagnostics/Netflow.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,82 @@
<default>0</default>
<Required>Y</Required>
</enable>
<history>
<FlowInterfaceTotals>
<_30 type="IntegerField">
<default>86400</default>
<Required>Y</Required>
<MinimumValue>0</MinimumValue>
<ValidationMessage>Must be an integer value greater than zero.</ValidationMessage>
</_30>
<_300 type="IntegerField">
<default>604800</default>
<Required>Y</Required>
<MinimumValue>0</MinimumValue>
<ValidationMessage>Must be an integer value greater than zero.</ValidationMessage>
</_300>
<_3600 type="IntegerField">
<default>2678400</default>
<Required>Y</Required>
<MinimumValue>0</MinimumValue>
<ValidationMessage>Must be an integer value greater than zero.</ValidationMessage>
</_3600>
<_86400 type="IntegerField">
<default>31536000</default>
<Required>Y</Required>
<MinimumValue>0</MinimumValue>
<ValidationMessage>Must be an integer value greater than zero.</ValidationMessage>
</_86400>
</FlowInterfaceTotals>
<FlowDstPortTotals>
<_300 type="IntegerField">
<default>3600</default>
<Required>Y</Required>
<MinimumValue>0</MinimumValue>
<ValidationMessage>Must be an integer value greater than zero.</ValidationMessage>
</_300>
<_3600 type="IntegerField">
<default>86400</default>
<Required>Y</Required>
<MinimumValue>0</MinimumValue>
<ValidationMessage>Must be an integer value greater than zero.</ValidationMessage>
</_3600>
<_86400 type="IntegerField">
<default>31536000</default>
<Required>Y</Required>
<MinimumValue>0</MinimumValue>
<ValidationMessage>Must be an integer value greater than zero.</ValidationMessage>
</_86400>
</FlowDstPortTotals>
<FlowSourceAddrTotals>
<_300 type="IntegerField">
<default>3600</default>
<Required>Y</Required>
<MinimumValue>0</MinimumValue>
<ValidationMessage>Must be an integer value greater than zero.</ValidationMessage>
</_300>
<_3600 type="IntegerField">
<default>86400</default>
<Required>Y</Required>
<MinimumValue>0</MinimumValue>
<ValidationMessage>Must be an integer value greater than zero.</ValidationMessage>
</_3600>
<_86400 type="IntegerField">
<default>31536000</default>
<Required>Y</Required>
<MinimumValue>0</MinimumValue>
<ValidationMessage>Must be an integer value greater than zero.</ValidationMessage>
</_86400>
</FlowSourceAddrTotals>
<FlowSourceAddrDetails>
<_86400 type="IntegerField">
<default>5356800</default>
<Required>Y</Required>
<MinimumValue>0</MinimumValue>
<ValidationMessage>Must be an integer value greater than zero.</ValidationMessage>
</_86400>
</FlowSourceAddrDetails>
</history>
</collect>
<activeTimeout type="IntegerField">
<default>1800</default>
Expand Down
6 changes: 4 additions & 2 deletions src/opnsense/scripts/netflow/flowd_aggregate.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,9 @@ def aggregate_flowd(config, do_vacuum=False):

# expire old data
for stream_agg_object in stream_agg_objects:
stream_agg_object.cleanup(do_vacuum)
stream_agg_class_name = type(stream_agg_object).__name__
history = config.history[stream_agg_class_name][stream_agg_object.resolution]
stream_agg_object.cleanup(history, do_vacuum)
del stream_agg_object
del metadata

Expand Down Expand Up @@ -192,7 +194,7 @@ def signal_handler(self, sig, frame):
if __name__ == '__main__':
# parse arguments and load config
parser = argparse.ArgumentParser()
parser.add_argument('--config', help='configuration yaml', default=None)
parser.add_argument('--config', help='configuration yaml', default="/usr/local/etc/flowd_aggregate.yaml")
parser.add_argument('--console', dest='console', help='run in console', action='store_true')
parser.add_argument('--profile', dest='profile', help='enable profiler', action='store_true')
parser.add_argument('--repair', dest='repair', help='init repair', action='store_true')
Expand Down
4 changes: 2 additions & 2 deletions src/opnsense/scripts/netflow/get_top_usage.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@

if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument('--config', '--config', help='configuration yaml', default=None)
parser.add_argument('--config', '--config', help='configuration yaml', default="/usr/local/etc/flowd_aggregate.yaml")
parser.add_argument('--provider', default='FlowInterfaceTotals')
parser.add_argument('--start_time', type=int, required=True)
parser.add_argument('--end_time', type=int, required=True)
Expand All @@ -54,7 +54,7 @@
# provider may specify multiple resolutions, we need to find the one most likely to serve the
# beginning of our timeframe
resolutions = sorted(agg_class.resolutions())
history_per_resolution = agg_class.history_per_resolution()
history_per_resolution = configuration.history[agg_class.__name__]

selected_resolution = resolutions[-1]
for resolution in resolutions:
Expand Down
24 changes: 23 additions & 1 deletion src/opnsense/scripts/netflow/lib/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
"""
import sys

SECONDS_PER_DAY = 60*60*24

def load_config(config_yaml=None):
""" setup configuration object
Expand All @@ -33,7 +34,7 @@ def load_config(config_yaml=None):
"""
if config_yaml:
import yaml
cnf_input = yaml.load(open(config_yaml, 'r'))
cnf_input = yaml.safe_load(open(config_yaml, 'r'))
else:
cnf_input = dict()

Expand All @@ -51,6 +52,27 @@ class Config(object):
flowd_source = '/var/log/flowd.log'
database_dir = '/var/netflow'
single_pass = False
history = {
'FlowInterfaceTotals': {
30: SECONDS_PER_DAY, # 24 hours
300: SECONDS_PER_DAY*7, # 7 days
3600: SECONDS_PER_DAY*31, # 31 days
86400: SECONDS_PER_DAY*365 # 365 days
},
'FlowDstPortTotals': {
300: 60*60, # 1 hour
3600: SECONDS_PER_DAY, # 24 hours
86400: SECONDS_PER_DAY*365 # 365 days
},
'FlowSourceAddrTotals': {
300: 60*60, # 1 hour
3600: SECONDS_PER_DAY, # 24 hours
86400: SECONDS_PER_DAY*365 # 365 days
},
'FlowSourceAddrDetails': {
86400: SECONDS_PER_DAY*62 # 62 days
}
}

def __init__(self, **kwargs):
for key in kwargs:
Expand Down
26 changes: 5 additions & 21 deletions src/opnsense/scripts/netflow/lib/aggregates/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,21 +49,6 @@ def resolutions(cls):
"""
return list()

@classmethod
def history_per_resolution(cls):
""" history to keep in seconds per sample resolution
:return: dict sample resolution / expire time (seconds)
"""
return dict()

@classmethod
def seconds_per_day(cls, days):
"""
:param days: number of days
:return: number of seconds
"""
return 60*60*24*days

def __init__(self, resolution, database_dir='/var/netflow'):
""" construct new flow sample class
:return: None
Expand Down Expand Up @@ -188,22 +173,21 @@ def add(self, flow):
# next start time
start_time += self.resolution

def cleanup(self, do_vacuum=False):
def cleanup(self, history, do_vacuum=False):
""" cleanup timeserie table
:param history: seconds of history to keep
:param do_vacuum: vacuum database
:return: None
"""
if self.is_db_open() and 'timeserie' in self._known_targets \
and self.resolution in self.history_per_resolution():
if self.is_db_open() and 'timeserie' in self._known_targets:
self._update_cur.execute('select max(mtime) as "[timestamp]" from timeserie')
last_timestamp = self._update_cur.fetchall()[0][0]
if type(last_timestamp) == datetime.datetime:
expire = self.history_per_resolution()[self.resolution]
expire_timestamp = last_timestamp - datetime.timedelta(seconds=expire)
expire_timestamp = last_timestamp - datetime.timedelta(seconds=history)
if last_timestamp > datetime.datetime.now():
# if data recorded seems to be in the future, use current timestamp for cleanup
# (prevent current data being removed)
expire_timestamp = datetime.datetime.now() - datetime.timedelta(seconds=expire)
expire_timestamp = datetime.datetime.now() - datetime.timedelta(seconds=history)

self._update_cur.execute('delete from timeserie where mtime < :expire', {'expire': expire_timestamp})
self.commit()
Expand Down
12 changes: 0 additions & 12 deletions src/opnsense/scripts/netflow/lib/aggregates/interface.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,18 +43,6 @@ def resolutions(cls):
# sample in 30 seconds, 5 minutes, 1 hour and 1 day
return [30, 300, 3600, 86400]

@classmethod
def history_per_resolution(cls):
"""
:return: dict sample resolution / expire time (seconds)
"""
return {
30: cls.seconds_per_day(1),
300: cls.seconds_per_day(7),
3600: cls.seconds_per_day(31),
86400: cls.seconds_per_day(365)
}

def __init__(self, resolution, database_dir='/var/netflow'):
"""
:param resolution: sample resolution (seconds)
Expand Down
13 changes: 0 additions & 13 deletions src/opnsense/scripts/netflow/lib/aggregates/ports.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,19 +42,6 @@ def resolutions(cls):
"""
return [300, 3600, 86400]

@classmethod
def history_per_resolution(cls):
"""
:return: dict sample resolution / expire time (seconds)
"""
# only save daily totals for a longer period of time, we probably only want to answer questions like
# "top usage over the last 30 seconds, 5 minutes, etc.."
return {
300: 3600,
3600: 86400,
86400: cls.seconds_per_day(365)
}

def __init__(self, resolution, database_dir='/var/netflow'):
"""
:param resolution: sample resolution (seconds)
Expand Down
22 changes: 0 additions & 22 deletions src/opnsense/scripts/netflow/lib/aggregates/source.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,19 +42,6 @@ def resolutions(cls):
"""
return [300, 3600, 86400]

@classmethod
def history_per_resolution(cls):
"""
:return: dict sample resolution / expire time (seconds)
"""
# only save daily totals for a longer period of time, we probably only want to answer questions like
# "top usage over the last 30 seconds, 5 minutes, etc.."
return {
300: 3600,
3600: 86400,
86400: cls.seconds_per_day(365)
}

def __init__(self, resolution, database_dir='/var/netflow'):
"""
:param resolution: sample resolution (seconds)
Expand Down Expand Up @@ -86,15 +73,6 @@ def resolutions(cls):
"""
return [86400]

@classmethod
def history_per_resolution(cls):
"""
:return: dict sample resolution / expire time (seconds)
"""
return {
86400: cls.seconds_per_day(62)
}

def __init__(self, resolution, database_dir='/var/netflow'):
"""
:param resolution: sample resolution (seconds)
Expand Down
1 change: 1 addition & 0 deletions src/opnsense/service/templates/OPNsense/Netflow/+TARGETS
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,4 @@ rc.conf.d:/etc/rc.conf.d/netflow
flowd.rc.conf.d:/etc/rc.conf.d/flowd
flowd_aggregate.rc.conf.d:/etc/rc.conf.d/flowd_aggregate
flowd.conf:/usr/local/etc/flowd.conf
flowd_aggregate.yaml:/usr/local/etc/flowd_aggregate.yaml
Loading