Getting Fortidragon to fly with rsyslog

[thetuxinator]

So i did a fork where i added my README/DOCS for Rsyslog. Maybe you can check if i missed something? See https://github.com/thetuxinator/fortinet-2-elasticsearch-rsyslog/blob/master/README-RSYSLOG.md maybe the "omelasticsearch-rule" is wrong?

regards

[enotspe]

I think we should follow datastreams index strategy

instead of

searchIndex="logstash-index"

I think we can use

searchIndex="logs-fortinet.fortigate"

Also, that could reuse the index templates configs.

Take into account that I split it even further, by type. if you can accomplish that with Rsyslog, that would make it completely transparent for current index templates.

[thetuxinator]

I think i followed https://chabik.com/2019/03/rsyslog-to-elasticsearch/ any idea on what to change in the config? I mean still if i understand it corretly it should at least find some feeded values from the elastic index, as rsyslog seems to send its stuff into when i activate it.

[thetuxinator]

Ok, i must be doing something completely wrong on the rsyslog rule. i am trying to figure out how to change the rule to not use logstash but do directly omelasticsearch

[thetuxinator]

looks like https://gist.github.com/pando85/75f06fb9a3b67788342e700fa8365674 could help

[thetuxinator]

I am giving it another try, as time allows now, however still can't get it to work. I think there is some documentation missing for the Elastic/Kibana Setup.

[enotspe]

I am giving it another try, as time allows now, however still can't get it to work. I think there is some documentation missing for the Elastic/Kibana Setup.

I am also checking the connection to Grafana Loki, so I am considering using Promtail for that, which might be more efficient because it is not java. We will see how that goes, I will keep eveybody updated.

[thetuxinator]

My main problem at the moment (as i am not yet experienced at Elastic/Kibana) is to get all the component templates inside of Elastic, is there a "nice" way to import them by the files in the repo?

best Regards

[enotspe]

mmmm nice is manual, it takes a while (30 min aprox) but you only got to do it once.

Just go into devtools inside kibana and load component templates both from Elastic ECS and FortiDragon specific. Do it manually one by one:

https://github.com/enotspe/fortinet-2-elasticsearch#on-kibana

[thetuxinator]

Tried already, if i try to add /master/index%20templates/component%20templates/logs-fortinet.forticlient%40ilm.json for example it seems to be incomplete and it claims about missing PUT or similar.

[enotspe]

You have to load the ilm policies first

[thetuxinator]

Tried already, if i try to add /master/index%20templates/component%20templates/logs-fortinet.forticlient%40ilm.json for example it seems to be incomplete and it claims about missing PUT or similar.

thx for your help, i created for example the "ilm policy fortigate.traffic" however still as mentioned above, cant do that example in the dev console:

{
"template": {
"mappings": {
"properties": {
"vcluster": {
"type": "long"
},
"iaid": {
"ignore_above": 1024,
"type": "keyword"
},
"infectedfilelevel": {
"type": "long"
},
"domainctrlip": {
"type": "ip"
},
"sentdelta": {
"type": "long"
},
"dstport": {
"type": "long"
},
"channel": {
"type": "long"
},
"mgmtcnt": {
"type": "long"
},
"sessionid": {
"ignore_above": 1024,
"type": "keyword"
},
"remote": {
"type": "ip"
},
"type": {
"type": "constant_keyword"
},
"ha_group": {
"type": "long"
},
"aptype": {
"type": "long"
},
"remport": {
"type": "long"
},
"virusid": {
"ignore_above": 1024,
"type": "keyword"
},
"translationid": {
"ignore_above": 1024,
"type": "keyword"
},
"cfgtid": {
"ignore_above": 1024,
"type": "keyword"
},
"scanned": {
"type": "long"
},
"countff": {
"type": "long"
},
"eapolcnt": {
"type": "long"
},
"portend": {
"type": "long"
},
"dstip": {
"type": "ip"
},
"nat": {
"type": "ip"
},
"identifier": {
"type": "long"
},
"vwpvlanid": {
"ignore_above": 1024,
"type": "keyword"
},
"newchannel": {
"type": "long"
},
"ip": {
"type": "ip"
},
"srcserver": {
"type": "long"
},
"fams_pause": {
"type": "long"
},
"ovrdid": {
"ignore_above": 1024,
"type": "keyword"
},
"received": {
"type": "long"
},
"botnetip": {
"type": "ip"
},
"vrf": {
"type": "long"
},
"centralnatid": {
"ignore_above": 1024,
"type": "keyword"
},
"scantime": {
"type": "long"
},
"chassisid": {
"ignore_above": 1024,
"type": "keyword"
},
"filteridx": {
"type": "long"
},
"nextstat": {
"type": "long"
},
"setuprate": {
"type": "long"
},
"countweb": {
"type": "long"
},
"remip": {
"type": "ip"
},
"alarmid": {
"ignore_above": 1024,
"type": "keyword"
},
"apstatus": {
"type": "long"
},
"assigned": {
"type": "ip"
},
"ha-prio": {
"type": "long"
},
"locip": {
"type": "ip"
},
"highcount": {
"type": "long"
},
"domainctrlauthstate": {
"type": "long"
},
"quotaused": {
"type": "long"
},
"stacount": {
"type": "long"
},
"countssh": {
"type": "long"
},
"epoch": {
"type": "long"
},
"filesize": {
"type": "long"
},
"oldchannel": {
"type": "long"
},
"sentpkt": {
"type": "long"
},
"countssl": {
"type": "long"
},
"local": {
"type": "ip"
},
"from_vcluster": {
"type": "long"
},
"ddnsserver": {
"type": "ip"
},
"policyid": {
"ignore_above": 1024,
"type": "keyword"
},
"sess_duration": {
"type": "long"
},
"sysuptime": {
"type": "long"
},
"encrypt": {
"type": "long"
},
"noise": {
"type": "long"
},
"oldchassisid": {
"ignore_above": 1024,
"type": "keyword"
},
"disklograte": {
"type": "long"
},
"countapp": {
"type": "long"
},
"infected": {
"type": "long"
},
"lanin": {
"type": "long"
},
"rssi": {
"type": "long"
},
"radioidclosest": {
"type": "long"
},
"column": {
"type": "long"
},
"portbegin": {
"type": "long"
},
"countdns": {
"type": "long"
},
"session_id": {
"ignore_above": 1024,
"type": "keyword"
},
"domainctrlauthtype": {
"type": "long"
},
"shapingpolicyid": {
"ignore_above": 1024,
"type": "keyword"
},
"countips": {
"type": "long"
},
"wanout": {
"type": "long"
},
"countemail": {
"type": "long"
},
"event_id": {
"ignore_above": 1024,
"type": "keyword"
},
"oldslot": {
"type": "long"
},
"quotamax": {
"type": "long"
},
"stage": {
"type": "long"
},
"appid": {
"ignore_above": 1024,
"type": "keyword"
},
"passedcount": {
"type": "long"
},
"dst_port": {
"type": "long"
},
"lease": {
"type": "long"
},
"rcvddelta": {
"type": "long"
},
"tranport": {
"type": "long"
},
"rcvdbyte": {
"type": "long"
},
"attackid": {
"ignore_above": 1024,
"type": "keyword"
},
"dstserver": {
"type": "long"
},
"shaperdroprcvdbyte": {
"type": "long"
},
"srcip": {
"type": "ip"
},
"newchassisid": {
"ignore_above": 1024,
"type": "keyword"
},
"eventtime": {
"type": "long"
},
"pid": {
"ignore_above": 1024,
"type": "keyword"
},
"used": {
"type": "long"
},
"vwlid": {
"ignore_above": 1024,
"type": "keyword"
},
"radioiddetected": {
"type": "long"
},
"xid": {
"ignore_above": 1024,
"type": "keyword"
},
"countav": {
"type": "long"
},
"crscore": {
"type": "long"
},
"newslot": {
"type": "long"
},
"limit": {
"type": "long"
},
"infectedfilesize": {
"type": "long"
},
"qtypeval": {
"type": "long"
},
"processtime": {
"type": "long"
},
"signal": {
"type": "long"
},
"auditid": {
"ignore_above": 1024,
"type": "keyword"
},
"assignip": {
"type": "ip"
},
"vulnid": {
"ignore_above": 1024,
"type": "keyword"
},
"audittime": {
"type": "long"
},
"transid": {
"ignore_above": 1024,
"type": "keyword"
},
"cfgtxpower": {
"type": "long"
},
"groupid": {
"ignore_above": 1024,
"type": "keyword"
},
"count": {
"type": "long"
},
"cpu": {
"type": "long"
},
"countdlp": {
"type": "long"
},
"transip": {
"type": "ip"
},
"freediskstorage": {
"type": "long"
},
"suspicious": {
"type": "long"
},
"to_vcluster": {
"type": "long"
},
"vcluster_member": {
"type": "long"
},
"craction": {
"type": "long"
},
"mtu": {
"type": "long"
},
"lowcount": {
"type": "long"
},
"disk": {
"type": "long"
},
"unit": {
"type": "long"
},
"wanin": {
"type": "long"
},
"port": {
"type": "long"
},
"used_for_type": {
"type": "long"
},
"proto": {
"type": "long"
},
"tunnelid": {
"ignore_above": 1024,
"type": "keyword"
},
"tunnelip": {
"type": "ip"
},
"countwaf": {
"type": "long"
},
"lanout": {
"type": "long"
},
"eventid": {
"ignore_above": 1024,
"type": "keyword"
},
"policy_id": {
"ignore_above": 1024,
"type": "keyword"
},
"fazlograte": {
"type": "long"
},
"slot": {
"type": "long"
},
"domainfilteridx": {
"type": "long"
},
"urlfilteridx": {
"type": "long"
},
"incidentserialno": {
"type": "long"
},
"duration": {
"type": "long"
},
"total": {
"type": "long"
},
"mem": {
"type": "long"
},
"rate": {
"type": "long"
},
"cat": {
"type": "long"
},
"checksum": {
"ignore_above": 1024,
"type": "keyword"
},
"countcifs": {
"type": "long"
},
"live": {
"type": "long"
},
"criticalcount": {
"type": "long"
},
"domainctrlprotocoltype": {
"type": "long"
},
"sentbyte": {
"type": "long"
},
"receivedsignature": {
"type": "long"
},
"radioid": {
"ignore_above": 1024,
"type": "keyword"
},
"mediumcount": {
"type": "long"
},
"transport": {
"type": "long"
},
"shaperdropsentbyte": {
"type": "long"
},
"malform_data": {
"type": "long"
},
"src_port": {
"type": "long"
},
"trueclntip": {
"type": "ip"
},
"locport": {
"type": "long"
},
"opertxpower": {
"type": "long"
},
"tranip": {
"type": "ip"
},
"serial": {
"type": "long"
},
"shaperperipdropbyte": {
"type": "long"
},
"expectedsignature": {
"type": "long"
},
"srcport": {
"type": "long"
},
"category": {
"type": "long"
},
"age": {
"type": "long"
},
"gateway": {
"type": "ip"
},
"rcvdpkt": {
"type": "long"
},
"totalsession": {
"type": "long"
},
"jitter": {
"type": "long"
},
"latency": {
"type": "long"
}
}
}
},
"_meta": {
"fortigate_version": "6.2.2"
}
}

[enotspe]

We can schedule a time for to help you online. Let me know if that would fine

[enotspe]

Hello @thetuxinator. We have moved all logic away from logstash to Elasticsearch ingest pipelines. That means that you can basically use any log collector/forwarder to receive logs from firewalls and deliver them to Elasticsearch. We use Elastic Agent with UDP input, but Rsyslog will work as well (probably better). Just point the logs to a datastream that calls the ingest pipeline. I will not test Rsyslog tough, I will test Vector but you can go ahead and try Rsyslog. Please share your results if you succeed.