Skip to content
This repository has been archived by the owner on Mar 8, 2024. It is now read-only.

Fresh install of hippocampe not working with Elasticsearch 5.3.x #68

Open
ag-michael opened this issue Oct 11, 2018 · 9 comments
Open

Fresh install of hippocampe not working with Elasticsearch 5.3.x #68

ag-michael opened this issue Oct 11, 2018 · 9 comments

Comments

@ag-michael
Copy link

Fresh install of hippocampe not working with Elasticsearch 5.3.x

Request Type

Bug

Work Environment

Question Answer
OS version (server) Redhat
OS version (client) 7.5
Hippocampe version / git hash f4d8807
Package Type From source
Browser type & version Curl

Problem Description

Unalbe to query shadowbook or enable sources due to the following error:

curl -XGET 10.x.x.x:5000/hippocampe/api/v1.0/shadowbook
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>500 Internal Server Error</title>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error and was unable to complete your request.  Either the server is overloaded or there is an error in the application.</p>

Hippocampe is being executed via app.py, it shows the following error:

[2018-10-11 23:19:12,046] ERROR in app: Exception on /hippocampe/api/v1.0/shadowbook [GET]
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "app.py", line 168, in shadowbookService
    if 'error' in reportJob:
TypeError: argument of type 'NoneType' is not iterable
10.4.6.106 - - [11/Oct/2018 23:19:12] "GET /hippocampe/api/v1.0/shadowbook HTTP/1.1" 500 -

Added a print_exc() on shadowbook.py#100 , which shows the following backtrace:

Traceback (most recent call last):
  File "/opt/Hippocampe/core/services/shadowbook.py", line 79, in initJob
    indexJob.createIndexJob()
  File "/opt/Hippocampe/core/services/modules/shadowbook/objects/IndexJob.py", line 71, in createIndexJob
    self.create()
  File "/opt/Hippocampe/core/services/modules/shadowbook/objects/Index.py", line 49, in create
    indexES.put_mapping(doc_type = self.typeNameES, body = self.docMapping)
  File "/usr/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 73, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/usr/lib/python2.7/site-packages/elasticsearch/client/indices.py", line 282, in put_mapping
    '_mapping', doc_type), params=params, body=body)
  File "/usr/lib/python2.7/site-packages/elasticsearch/transport.py", line 318, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "/usr/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 128, in perform_request
    self._raise_error(response.status, raw_data)
  File "/usr/lib/python2.7/site-packages/elasticsearch/connection/base.py", line 124, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
RequestError: TransportError(400, u'illegal_argument_exception', u'mapper [status] cannot be changed from type [text] to [keyword]')

This is likely caused by https://github.com/TheHive-Project/Hippocampe/blob/master/core/services/modules/shadowbook/objects/IndexJob.py#L51

Steps to Reproduce

  1. Follow the Install instructions
  2. run 'python core/app.py'
  3. Curl an api url such as 'curl -XGET 127.0.0.1:5000/hippocampe/api/v1.0/shadowbook'
@ninSmith
Copy link
Contributor

Hi,

Try with ES 5.1 and see if it works

@ag-michael
Copy link
Author

@ninSmith TheHive apparently needs at least ES 5.6 https://github.com/TheHive-Project/TheHiveDocs/blob/master/installation/install-guide.md#build-it-yourself

Not only that, I need to run a Kibana front end that supports csv exporting (even 5.6 does not support this). I can only upgrade at this point, can't downgrade. If you're wanting to see if the problem will get resolved this way, I can test Hippocampe with a test ES instance and let you know the results.

@kx499-zz
Copy link

I had the same issue on 5.2 - resolved it by nuking the hippocampe index and starting over

@kx499-zz
Copy link

I think the trick is to not create a default index in kibana until you run shadowbook and it completes. Honestly not sure though. Somehow the .kibana index and hippocampe index cause conflicts in the mappings. My best guess, not an ES guru by any stretch

@kx499-zz
Copy link

Additionally, if you want to use visualizations in kibana you need to update field types for description for search, source, and dashboard types. here's the example for search.

PUT /_all/_mapping/search?update_all_types
{
"properties": {
"description": {
"type": "text"
}
}
}

@ag-michael
Copy link
Author

@kx499 how can I nuke the hippocampe index? I'm new to ES.

Visualizations are not important for me, but I do need hippocampe to use the ES instance as Kibana.

@kx499-zz
Copy link

curl -X DELETE "localhost:9200/hippocampe"

@ag-michael
Copy link
Author

@ninSmith ES 5.6 works for me. However, I had to setup a dedicated ES instance. I wasn't able to test with Kibana.

Please leave this issue open, I will try to upgrade production ES to 5.6, get rid of kibana ,setup hippocampe and then reinstall Kibana.

@ag-michael
Copy link
Author

I wanted to update and say I won't be having any free time soon to pursue troubleshooting this issue.
I'll leave it open in case the project is still interested in resolving this.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@kx499-zz @ninSmith @ag-michael