Intermittent Issue with IndexError

[ghost]

Thank you so much for posting this! I saw a recording of the demo at Akamai Edge and I'm excited the code has been published. I am using this to build a slightly simpler Docker container (no Logstash, just pushing out the event via TCP to our existing logstash server) and I've tweaked the python a tiny bit to run in Python 3.

I am running into an intermittent issue with the python script and I wanted to see if you or anyone else who has used the script has run into the same thing. Every now and then, the script will fail with an IndexError at the following point:

Traceback (most recent call last):
   File "/tmp/siem_1.py", line 64, in <module>
      rules_array[i][member_as_singular] = base64.b64decode(item).decode('UTF-8')
IndexError: list index out of range

This happens both when I'm running in Docker and when I run the python script directly on my laptop. The script usually runs fine on it's next scheduled run, so I'm not really missing out on any logs, but I don't know if this is to be expected or if I should maybe wrap this bit in a try/except.

[dsztykman]

Could you post the Json line in question when this happens? I assume it’s because the array contains only a single Rule and I expect multiple

[ghost]

Here are a few example events that produced the index error:

{
   'httpMessage':{
      'status':'403',
      'host':'www.OURDOMAIN.com',
      'responseHeaders':'Server%3a%20AkamaiGHost%0d%0aMime-Version%3a%201.0%0d%0aContent-Type%3a%20text%2fhtml%0d%0aContent-Length%3a%20270%0d%0aExpires%3a%20Fri,%2006%20Jul%202018%2018%3a42%3a37%20GMT%0d%0aDate%3a%20Fri,%2006%20Jul%202018%2018%3a42%3a37%20GMT%0d%0aConnection%3a%20close%0d%0a',
      'tls':'tls1.2',
      'protocol':'HTTP/1.0',
      'method':'POST',
      'bytes':'270',
      'port':'443',
      'requestHeaders':'Accept%3a%20*%2f*%0d%0aUser-Agent%3a%20Mozilla%2f5.0%20(Windows%20NT%2010.0%3b%20WOW64%3b%20rv%3a45.0)%20Gecko%2f20100101%20Firefox%2f45.0%0d%0aReferer%3a%20https%3a%2f%2fwww.OURDOMAIN.com%2findex.cfm%3ffa%3dMain.Comments%0d%0aContent-Type%3a%20application%2fx-www-form-urlencoded%0d%0aHost%3a%20www.OURDOMAIN.com%0d%0aContent-Length%3a%208015%0d%0a',
      'path':'/&',
      'requestId':'3d17f99',
      'start':'1530902557'
   },
   'attackData':{
      'ruleSelectors':'QVJHUzpiaWRkZXJDb21tZW50cw%3d%3d%3bQVJHUzpiaWRkZXJDb21tZW50cw%3d%3d%3bQVJHUzpiaWRkZXJDb21tZW50cw%3d%3d%3bQVJHUzpiaWRkZXJDb21tZW50cw%3d%3d%3bQVJHUzpiaWRkZXJDb21tZW50cw%3d%3d%3bQVJHUzpiaWRkZXJDb21tZW50cw%3d%3d%3bQVJHUzpiaWRkZXJDb21tZW50cw%3d%3d%3bQVJHUzpiaWRkZXJDb21tZW50cw%3d%3d%3bQVJHUzpiaWRkZXJDb21tZW50cw%3d%3d%3b',
      'ruleTags':'T1dBU1BfQ1JTL1dFQl9BVFRBQ0svU1FMX0lOSkVDVElPTg%3d%3d%3bT1dBU1BfQ1JTL1dFQl9BVFRBQ0svWFNT%3bT1dBU1BfQ1JTL1dFQl9BVFRBQ0svWFNT%3bT1dBU1BfQ1JTL1dFQl9BVFRBQ0svWFNT%3bT1dBU1BfQ1JTL1dFQl9BVFRBQ0svWFNT%3bT1dBU1BfQ1JTL1dFQl9BVFRBQ0svU1BFQ0lBTF9DSEFSUw%3d%3d%3bT1dBU1BfQ1JTL1dFQl9BVFRBQ0svU1FMX0lOSkVDVElPTg%3d%3d%3bT1dBU1BfQ1JTL1dFQl9BVFRBQ0svU1FMX0lOSkVDVElPTg%3d%3d%3bT1dBU1BfQ1JTL1dFQl9BVFRBQ0svU1FMX0lOSkVDVElPTg%3d%3d%3bQUtBTUFJL1BPTElDWS9TUUxfSU5KRUNUSU9OX0FOT01BTFk%3d',
      'ruleMessages':'U1FMIEluamVjdGlvbiBBdHRhY2s%3d%3bUG9zc2libGUgWFNTIEF0dGFjayBEZXRlY3RlZCAtIEhUTUwgVGFnIEhhbmRsZXI%3d%3bWFNTIEF0dGFjayBEZXRlY3RlZA%3d%3d%3bSUUgWFNTIEZpbHRlcnMgLSBBdHRhY2sgRGV0ZWN0ZWQ%3d%3bSUUgWFNTIEZpbHRlcnMgLSBBdHRhY2sgRGV0ZWN0ZWQ%3d%3bUmVzdHJpY3RlZCBTUUwgQ2hhcmFjdGVyIEFub21hbHkgRGV0ZWN0aW9uIEFsZXJ0IC0gVG90YWwgIyBvZiBzcGVjaWFsIGNoYXJhY3RlcnMgZXhjZWVkZWQ%3d%3bQ2xhc3NpYyBTUUwgSW5qZWN0aW9uIFByb2JlcyAyLzI%3d%3bQ2hhaW5lZCBTUUwgSW5qZWN0aW9uIEF0dGVtcHRzIDIvMg%3d%3d%3bU1FMIFNFTEVDVCBTdGF0ZW1lbnQgQW5vbWFseSBEZXRlY3Rpb24gQWxlcnQ%3d%3bQW5vbWFseSBTY29yZSBFeGNlZWRlZCBmb3IgU1FMIEluamVjdGlvbg%3d%3d',
      'ruleVersions':'NA%3d%3d%3bNA%3d%3d%3bNA%3d%3d%3bNA%3d%3d%3bNA%3d%3d%3bMw%3d%3d%3bMw%3d%3d%3bNA%3d%3d%3bMg%3d%3d%3bMQ%3d%3d',
      'clientIP':'193.201.224.246',
      'configId':'XXXX',
      'ruleData':'IEF2ZW5nZXJzIDxh%3bPGEg%3baHJlZj0%3d%3bJyksIEQ%3d%3bJyksIEQ%3d%3bIg%3d%3d%3bIiBhbmQgYWZ0ZXIgYnkgTWVuL1NwaWRlciB3b3cgZW5kZWQgdXAuDQoNCmEgdW5pcXVlIGlyb25tYW4gbW90aW9uIGhhZCBiZWVuIHRoZSBmaXJzdCBhIHRob3VnaHQgd2l0aCAoeW91IGtub3csIGkgc3VwcG9zZSBiYXRtYW4gY29tbWVuY2VzIHB1dCBkb3duIHRoZSBmb290IHdvcmsgZm9yIGNoaWxkcmVuIG9mIG1lbikuIHRoZSBkZXZpY2UgcHJvZHVjZWQgZXZlcnkgc2luZ2xlIGluIG9yZGVyIHRvIHdvcmsgd2VsbC4gZXZlcnkgY29taWMgYm9vayBwbGF5ZXIgZm9ybWVyIHdhcyBtYWRlIHByb3Blcmx5IHNlcXVlbCB3b3VsZCd2ZSBvbmx5IGNvbWUgYWJvdXQgZnJvbSB0aGUgMXN0IG1hZGUgcHJvZml0cy4gY29tYmluZWQgd2l0aCBiYXRtYW4sIG1hbnkgcGVvcGxlIGJlbGlldmVkIGl0IHdhcyBnb2luZyB0byBwbGF5IGluIGxpbmUgd2l0aCBzb21ldGhpbmcgaGFyZGVyLCBpdCBpcm9uIG1hbiBtdXN0IGJlIGdvb2QgdG8gZ2V0IHRoZSBiYWxsIHJvbGxpbmcuDQoNCkZ1Y2tDYXphZG9ycyA4%3bIGNhc2UgaWYgd2hhdCB0aGlzIG1lZGljYWwgcHJvZmVzc2lvbnNhbCBzYXlzIGlzIDEwMCUgc3BlY2lmaWMsIHRoaXMgaGFzIGJlZW4gYSBub3Qgc21hcnQgaW1wb3J0YW50IGl0ZW0gYXMgVHJ1bXAgdG8gYWNjb21wbGlzaC4gb2YgdGhlIGNvcnBvcmFsIHN0YXlpbmcgb2YgYW4gYXBwbGljYW50IHNob3VsZG4gYmUgbWFkZSBpbnRvIGEgY3J1c2FkZSBhZCB0aGF0J3MgdGhlIHRydXRoIG1pZ2h0IGJlIGFwcGFyZW50bHkgd2hhdCB3ZW50IGRvd24gcG9zdC4gaW4sIHRoaXMgcGVyc29uIHdpbGwgbmVlZCBzdWZmZXIgaGlzIGxpY2VuY2Ugc28gbG9uZyBhcyBoZSB3aXRoIHlvdXIga25vd2xlZGdlIGZpeGVkIHNvbWUgdGhpbmcgaGUgd2FzIGNvbmZpZGVudCB0byBiZSA8YSBocmVmPWh0dHBzOi8vd3d3LmNydW5jaGJhc2UuY29tL29yZ2FuaXphdGlvbi9sYXRhbWRhdGUtY29tPmxhdGFtZGF0ZSBzY2FtPC9hPiB3cm9uZyBhbmQgYWxzbyB0aGF0IGhlIHdvdWxkbid0IGRyYWZ0IGhpbXNlbGYuIGluY2x1ZGluZyB3aGF0IHRoZSBoZWNrIGlzIHByZXNlbnRlZCBpbiBmaWxtIHByb2R1Y3Rpb24gY29tcGFueTogYnkgbWVhbnMgb2YgVGhvciB3aXRoIHRoZSB1c2Ugb2YgU3VwIHRvcm1icmVha2VyIHVuZG91YnRlZGx5IHN0YWIgVGhhbm9zIG1ha2luZyBhbiBhdHRlbXB0LCBpdCBkZWZpbml0ZWx5IGZlZWxzIFRob3IgYXMgd2VsbCBhcyBTdG9ybWJyZWFrZXIgaXMgY29uc2lkZXJlZCB0byBiZSBtdWNoIHN0cm9uZ2VyIHNpbmNlIGZyb20gYW9zIE1DVS4gYmVhciBpbiBtaW5kLCB0b3RhbGx5IGZyb20gdGhlIGJlc3Qgb3V0bG9vayB3aGljaCBpdCBkb2VzbiBwcmV0dHkgbXVjaCBiZSB0aGUgYmV0dGVyIGNob2ljZS4gdGhlc2UgZ3V5cyByZWNvcmRlZCBpbmNyZWFzZSB0aGUgbm90aW9uOiAiYXNzdW1pbmcgdGhhdCBUaGFub3MgYmVjb21lcyB0aGUgZW50aXJlIGluZmluaXR5IGdlbXN0b25lcyBoZSBjb3VsZCBiZSB0aGUgbW9zdCBwb3dlcmZ1bCBmaW5kaW5nIHlvdXJzZWxmIGluIHRoZSB3b3JsZCwgYWZ0ZXIgVGhvciBzdHJvbGxpbmcgdG8gdGhlIHBlYWsgYW5kIGluZmxpY3RzIGEgdGhyaWxsaW5nIHdvdW5kIHdoaWNoIGNhbiBUaGFub3MuIGlmIG9yIHdoZW4g%3bVG90YWwgbnVtYmVyIG9mIFNRTCBzdGF0ZW1lbnQgZWxlbWVudHM6IDU%3d%3bVmVjdG9yIFNjb3JlOiAxOSwgREVOWSB0aHJlc2hvbGQ6IDE0LCBBbGVydCBSdWxlczogOTUwOTAxOjk3MzMwMDo5NzMzMDQ6OTczMzMzOjk3MzMzNTo5ODExNzM6OTgxMjQzOjk4MTI0OTo5ODEzMDAsIERlbnkgUnVsZTogLCBMYXN0IE1hdGNoZWQgTWVzc2FnZTogU1FMIFNFTEVDVCBTdGF0ZW1lbnQgQW5vbWFseSBEZXRlY3Rpb24gQWxlcnQ%3d',
      'policyId':'XXXXXX',
      'ruleActions':'YWxlcnQ%3d%3bYWxlcnQ%3d%3bYWxlcnQ%3d%3bYWxlcnQ%3d%3bYWxlcnQ%3d%3bYWxlcnQ%3d%3bYWxlcnQ%3d%3bYWxlcnQ%3d%3bYWxlcnQ%3d%3bZGVueQ%3d%3d',
      'rules':'OTUwOTAx%3bOTczMzAw%3bOTczMzA0%3bOTczMzMz%3bOTczMzM1%3bOTgxMTcz%3bOTgxMjQz%3bOTgxMjQ5%3bOTgxMzAw%3bU1FMLUlOSkVDVElPTi1BTk9NQUxZ'
   },
   'format':'json',
   'version':'1.0',
   'geo':{
      'asn':'25092',
      'city':'KIEV',
      'country':'UA',
      'continent':'EU',
      'regionCode':''
   },
   'type':'akamai_siem'
}

{
   'httpMessage':{
      'status':'403',
      'host':'www.OURDOMAIN.com',
      'responseHeaders':'Server%3a%20AkamaiGHost%0d%0aMime-Version%3a%201.0%0d%0aContent-Type%3a%20text%2fhtml%0d%0aContent-Length%3a%20270%0d%0aExpires%3a%20Fri,%2006%20Jul%202018%2018%3a44%3a01%20GMT%0d%0aDate%3a%20Fri,%2006%20Jul%202018%2018%3a44%3a01%20GMT%0d%0aConnection%3a%20close%0d%0a',
      'tls':'tls1.2',
      'protocol':'HTTP/1.0',
      'method':'POST',
      'bytes':'270',
      'port':'443',
      'requestHeaders':'Accept%3a%20*%2f*%0d%0aUser-Agent%3a%20Mozilla%2f5.0%20(Windows%20NT%205.0%3b%20rv%3a30.1)%20Gecko%2f20100101%20Firefox%2f30.1%0d%0aReferer%3a%20https%3a%2f%2fwww.OURDOMAIN.com%2findex.cfm%3ffa%3dMain.Comments%0d%0aContent-Type%3a%20application%2fx-www-form-urlencoded%0d%0aHost%3a%20www.OURDOMAIN.com%0d%0aContent-Length%3a%201938%0d%0aCookie%3a%20ARRAffinity%3d3e5d826c0cbf6e8bcfe27235d10e21d79d43787163256402d1c4dffa42a501ea%3b%20CFID%3d41773453%3b%20CFTOKEN%3d56340747%3b%20COUNTRY%3d%22%22%0d%0a',
      'path':'/&',
      'requestId':'3d3d8c7',
      'start':'1530902641'
   },
   'attackData':{
      'ruleSelectors':'QVJHUzpiaWRkZXJDb21tZW50cw%3d%3d%3bQVJHUzpiaWRkZXJDb21tZW50cw%3d%3d%3bQVJHUzpiaWRkZXJDb21tZW50cw%3d%3d%3bQVJHUzpiaWRkZXJDb21tZW50cw%3d%3d%3bQVJHUzpiaWRkZXJDb21tZW50cw%3d%3d%3b',
      'ruleTags':'T1dBU1BfQ1JTL1dFQl9BVFRBQ0svU1FMX0lOSkVDVElPTg%3d%3d%3bT1dBU1BfQ1JTL1dFQl9BVFRBQ0svWFNT%3bT1dBU1BfQ1JTL1dFQl9BVFRBQ0svWFNT%3bT1dBU1BfQ1JTL1dFQl9BVFRBQ0svU1BFQ0lBTF9DSEFSUw%3d%3d%3bT1dBU1BfQ1JTL1dFQl9BVFRBQ0svU1FMX0lOSkVDVElPTg%3d%3d%3bQUtBTUFJL1BPTElDWS9YU1NfQU5PTUFMWQ%3d%3d',
      'ruleMessages':'U1FMIEluamVjdGlvbiBBdHRhY2s%3d%3bUG9zc2libGUgWFNTIEF0dGFjayBEZXRlY3RlZCAtIEhUTUwgVGFnIEhhbmRsZXI%3d%3bWFNTIEF0dGFjayBEZXRlY3RlZA%3d%3d%3bUmVzdHJpY3RlZCBTUUwgQ2hhcmFjdGVyIEFub21hbHkgRGV0ZWN0aW9uIEFsZXJ0IC0gVG90YWwgIyBvZiBzcGVjaWFsIGNoYXJhY3RlcnMgZXhjZWVkZWQ%3d%3bU1FMIFNFTEVDVCBTdGF0ZW1lbnQgQW5vbWFseSBEZXRlY3Rpb24gQWxlcnQ%3d%3bQW5vbWFseSBTY29yZSBFeGNlZWRlZCBmb3IgQ3Jvc3MtU2l0ZSBTY3JpcHRpbmc%3d',
      'ruleVersions':'NA%3d%3d%3bNA%3d%3d%3bNA%3d%3d%3bMw%3d%3d%3bMg%3d%3d%3bMQ%3d%3d',
      'clientIP':'193.201.224.246',
      'configId':'XXXXX',
      'ruleData':'IGhvdXNlcyA8YQ%3d%3d%3bPGEg%3baHJlZj0%3d%3bOg%3d%3d%3bVG90YWwgbnVtYmVyIG9mIFNRTCBzdGF0ZW1lbnQgZWxlbWVudHM6IDM%3d%3bVmVjdG9yIFNjb3JlOiAxMCwgREVOWSB0aHJlc2hvbGQ6IDksIEFsZXJ0IFJ1bGVzOiA5NTA5MDE6OTczMzAwOjk3MzMwNDo5ODExNzM6OTgxMzAwLCBEZW55IFJ1bGU6ICwgTGFzdCBNYXRjaGVkIE1lc3NhZ2U6IFNRTCBTRUxFQ1QgU3RhdGVtZW50IEFub21hbHkgRGV0ZWN0aW9uIEFsZXJ0',
      'policyId':'XXXXXXXX',
      'ruleActions':'YWxlcnQ%3d%3bYWxlcnQ%3d%3bYWxlcnQ%3d%3bYWxlcnQ%3d%3bYWxlcnQ%3d%3bZGVueQ%3d%3d',
      'rules':'OTUwOTAx%3bOTczMzAw%3bOTczMzA0%3bOTgxMTcz%3bOTgxMzAw%3bWFNTLUFOT01BTFk%3d'
   },
   'format':'json',
   'version':'1.0',
   'geo':{
      'asn':'25092',
      'city':'KIEV',
      'country':'UA',
      'continent':'EU',
      'regionCode':''
   },
   'type':'akamai_siem'
}