Updated README.md for BigQuery to Elasticsearch template #298
Conversation
Updated README.md for BigQuery to Elasticsearch template
|
|
||
| Pipeline flow is illustrated below: | ||
|
|
||
|  |
There was a problem hiding this comment.
Image doesn't reflect the existing flow as current template has only 3 steps
| ### Parameters | ||
|
|
||
| The template requires the following parameters: | ||
| * inputTableSpec: Table in BigQuery to read from in form of: my-project:my-dataset.my-table. Either this or query must be provided. |
There was a problem hiding this comment.
inputTableSpec is also optional parameter since user can supply either query or inputTableSpec
| @@ -61,7 +82,7 @@ echo '{ | |||
| "image":"'${TARGET_GCR_IMAGE}'", | |||
| "metadata":{ | |||
| "name":"BigQuery to Elasticsearch", | |||
| "description":"Replicates BigQuery data into an Elasticsearch index", | |||
| "description":"Ingests BigQuery data into an Elasticsearch index", | |||
| "parameters":[ | |||
There was a problem hiding this comment.
For below parameters isOptional should be changed to true
inputTableSpec, elasticsearchUsername, elasticsearchPassword
ApiKey should be added to Json spec as mandatory parameter
There was a problem hiding this comment.
Hi @prathapreddy123,
I want to use this template by using the username and password and not use the API key. Are you aware of any approach in which I may be able to do it.
There was a problem hiding this comment.
If you would like to use username and password you can use elasticsearchUsername and elasticsearchPassword and pass dummy value to apiKey parameter. In the cases where username and password is provided along with apikey, username and password will take precedence
There was a problem hiding this comment.
Hi @prathapreddy123 ,
I have already tried by passing the : as base64encoded string and also passing the username and password in the username and password field but still the job/pipeline fails to execute. Can you provide me with a sample glcoud command used to create the job using this template as you are suggesting.
There was a problem hiding this comment.
@prathapreddy123 Thanks I got it working by just passing a simple single space as apikey and passing the username and password.
|
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
|
Hi @suyograo -- thanks for your contribution here. Same thing - we unfortunately ran behind on reviews, and the structure of the project changed a bit. Is there still interest in pursuing this change? We can work together on a plan to resolve conflicts + review the docs. |
Updated README.md for BigQuery to Elasticsearch template