Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

python3 conversion, url checking methods #1

Open
wants to merge 26 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
9dfa126
convert to alpine 3.9 and python3
eapearson Feb 17, 2019
34624cb
more python3 fixes
eapearson Feb 17, 2019
049c3ed
a few stragglers...
eapearson Feb 17, 2019
43a99c0
add methods for checking image and html urls; refactored tests into o…
eapearson Feb 25, 2019
d500d53
add timeout to url checkers
eapearson Feb 26, 2019
0feef92
updating docs
eapearson Mar 19, 2019
5320c30
Merge remote-tracking branch 'eap/master'
eapearson Mar 19, 2019
05b95a2
add test runner script tweak desc
eapearson Mar 19, 2019
ce88250
fix tests for url checks
eapearson Mar 22, 2019
a9804ef
add compile back to install
eapearson Mar 22, 2019
721223a
Merge pull request #1 from eapearson/master
eapearson Mar 22, 2019
2f2473e
use legacy chown
eapearson Mar 29, 2019
98c3581
Merge pull request #2 from eapearson/master
eapearson Mar 29, 2019
99ca5d1
remove config validation of mongo settings
eapearson Mar 29, 2019
dbf59d7
Merge pull request #3 from eapearson/master
eapearson Mar 29, 2019
453beba
remove all alert functionality and adjust tests
eapearson Mar 30, 2019
6b30e3c
tweak readme
eapearson Mar 30, 2019
7fd7fb5
Merge pull request #4 from eapearson/master
eapearson Mar 30, 2019
28ffd7d
handle redirect (301, 302)
eapearson Mar 30, 2019
f42a54f
Merge pull request #5 from eapearson/master
eapearson Mar 30, 2019
7b35899
ignore _attic, handy for stashing in during development
eapearson Sep 25, 2020
b08fa7e
update alpine base, and python deps.
eapearson Sep 25, 2020
50ddbb7
improve readme; add self signed cert example
eapearson Sep 25, 2020
a753d7d
add support for user-agent header
eapearson Sep 25, 2020
438d623
add tests to test user-agent presence, absence, and format.
eapearson Sep 25, 2020
6a45e8e
Merge pull request #6 from eapearson/master
eapearson Sep 25, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions .coveragerc
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
[run]
omit =
# omit code not under our control
UIService/UIServiceServer.py
UIService/authclient.py
UIService/__init__.py

[report]
omit =
# omit code not under our control
UIService/UIServiceServer.py
UIService/authclient.py
UIService/__init__.py
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,4 @@ test_local
sdk.cfg
*.pyc
*.bak-*
_attic
4 changes: 4 additions & 0 deletions .vscode/settings.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{
"python.linting.pylintEnabled": true,
"python.pythonPath": "/opt/local/bin/python3.6"
}
90 changes: 46 additions & 44 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,68 +1,70 @@
FROM alpine:3.8 as builder
FROM alpine:3.12 as builder
MAINTAINER KBase Developer

# The build stage needs just enough to run the KBase SDK tools.
#

# update system and system dependencies
RUN apk upgrade --update-cache --available \
&& apk add --update --no-cache \
apache-ant=1.10.4-r0 \
bash=4.4.19-r1 \
git=2.18.0-r0 \
linux-headers=4.4.6-r2 \
make=4.2.1-r2 \
openjdk8=8.171.11-r0 \
python2=2.7.15-r0

RUN mkdir -p /kb \
&& git clone --depth=1 https://github.com/kbase/kb_sdk /kb/kb_sdk \
&& cd /kb/kb_sdk \
&& make
RUN apk upgrade --update-cache --available && \
apk add --update --no-cache apache-ant bash git linux-headers make openjdk8

# Currently need this special install for python3. oh, python.
RUN apk add --no-cache python3 && \
python3 -m ensurepip && \
rm -r /usr/lib/python*/ensurepip && \
pip3 install --upgrade pip setuptools && \
if [ ! -e /usr/bin/pip ]; then ln -s pip3 /usr/bin/pip ; fi && \
if [[ ! -e /usr/bin/python ]]; then ln -sf /usr/bin/python3 /usr/bin/python; fi && \
rm -r /root/.cache

RUN mkdir -p /kb && \
git clone --depth=1 https://github.com/kbase/kb_sdk /kb/kb_sdk && \
cd /kb/kb_sdk && \
make

COPY ./ /kb/module

RUN mkdir -p /kb/module/work/cache \
&& chmod -R a+rw /kb/module \
&& cd /kb/module \
&& PATH=$PATH:/kb/kb_sdk/bin make install
RUN mkdir -p /kb/module/work/cache && \
chmod -R a+rw /kb/module && \
cd /kb/module && \
PATH=$PATH:/kb/kb_sdk/bin make install

# Final image

FROM alpine:3.8
FROM alpine:3.12
MAINTAINER KBase Developer

# update system and system dependencies
RUN apk upgrade --update-cache --available \
&& apk add --update --no-cache \
bash=4.4.19-r1 \
g++=6.4.0-r8 \
git=2.18.0-r0 \
libffi-dev=3.2.1-r4 \
linux-headers=4.4.6-r2 \
make=4.2.1-r2 \
openssl-dev=1.0.2o-r2 \
py2-pip=10.0.1-r0 \
python2=2.7.15-r0 \
python2-dev=2.7.15-r0
RUN apk upgrade --update-cache --available && \
apk add --update --no-cache bash g++ git libffi-dev linux-headers make openssl-dev

# Currently need this special install for python3. oh, python.
RUN apk add --no-cache python3 python3-dev py3-setuptools && \
python3 -m ensurepip && \
rm -r /usr/lib/python*/ensurepip && \
pip3 install --upgrade pip setuptools && \
if [ ! -e /usr/bin/pip ]; then ln -s pip3 /usr/bin/pip ; fi && \
if [[ ! -e /usr/bin/python ]]; then ln -sf /usr/bin/python3 /usr/bin/python; fi && \
rm -r /root/.cache

# install python dependencies for the service runtime.
RUN pip install --upgrade pip && \
pip install \
cffi==1.11.5 \
jinja2==2.10 \
jsonrpcbase==0.2.0 \
ndg-httpsclient==0.5.1 \
pymongo===3.7.1 \
python-dateutil==2.7.3 \
pytz==2018.5 \
requests==2.19.1 \
uwsgi==2.0.17.1

cffi==1.14.3 \
coverage==4.5.4 \
jinja2==2.11.2 \
jsonrpcbase==0.2.0 \
ndg-httpsclient==0.5.1 \
git+https://github.com/nose-devs/nose@master \
python-dateutil==2.8.1 \
pytz==2020.1 \
requests==2.24.0 \
uwsgi==2.0.19.1

RUN addgroup --system kbmodule && \
adduser --system --ingroup kbmodule kbmodule

COPY --from=builder --chown=kbmodule:kbmodule /kb/module /kb/module
COPY --from=builder /kb/module /kb/module
RUN chown -R kbmodule:kbmodule /kb/module

WORKDIR /kb/module

Expand Down
4 changes: 3 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,9 @@ build-test-script:
@echo 'export KB_AUTH_TOKEN=`cat /kb/module/work/token`' >> $(TEST_DIR)/$(TEST_SCRIPT_NAME)
@echo 'export PYTHONPATH=$$script_dir/../$(LIB_DIR):$$PATH:$$PYTHONPATH' >> $(TEST_DIR)/$(TEST_SCRIPT_NAME)
@echo 'cd $$script_dir/../$(TEST_DIR)' >> $(TEST_DIR)/$(TEST_SCRIPT_NAME)
@echo 'python -m nose --with-coverage --cover-package=$(SERVICE_CAPS) --cover-html --cover-html-dir=/kb/module/work/test_coverage --nocapture --nologcapture .' >> $(TEST_DIR)/$(TEST_SCRIPT_NAME)
@echo 'echo "Running tests"' >> $(TEST_DIR)/$(TEST_SCRIPT_NAME)
@echo 'echo `pwd`' >> $(TEST_DIR)/$(TEST_SCRIPT_NAME)
@echo 'python -m nose --with-coverage --cover-package=$(SERVICE_CAPS) --cover-html --cover-html-dir=/kb/module/work/test_coverage --nocapture --nologcapture --cover-config-file=$$script_dir/../$(TEST_DIR)/coverage.cfg .' >> $(TEST_DIR)/$(TEST_SCRIPT_NAME)
@chmod +x $(TEST_DIR)/$(TEST_SCRIPT_NAME)

docker-image-dev:
Expand Down
81 changes: 32 additions & 49 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,32 +1,48 @@

# UIService

---

A [KBase](https://kbase.us) module generated by the [KBase SDK](https://github.com/kbase/kb_sdk).

## Local development
## Using ui-service

See the spec file, UIService.spec, for api and type definitions.

## Getting started with testing and development

See [Developer Docs](docs/development.md)

old way:
## Tests

## Local development with kbase-ui

To make the image locally:

```
PATH=$PATH:`pwd`/../../kb_sdk/bin
kb-sdk test
bash scripts/run-bash.sh
bash scripts/start_server.sh
make docker-image-dev
```

new way:
Then to run it:

```
make docker-image-dev
make run-docker-image-dev
```

This will start the container attached to the Docker custom network used by kbase-ui.

Then in kbase-ui, start it with the dynamic-services option, like

```
kbase-ui start-dev env=dev build=dev build-image=f dynamic-services="ui-service"
```

## Pure local image build

kb_sdk's ModuleTester.java contains the code to build the docker image. All it does, as far as building goes, is construct a simple docker build bash command string. (It also manages images, etc.)

```
docker build --rm -t TARGET_IMAGE_NAME REPO_PATH
```

where TARGET_IMAGE_NAME is test/MODULE:latest

Expand All @@ -40,47 +56,14 @@ docker build --rm -t test/ui-service:dev

If you are starting from a cold docker cache the initial build may take several minutes: the kbase base image is rather huge.

## Actually...

To make the image locally:

```
make docker-image-dev
```

Then to run it:

```
make run-docker-image-dev
```

This will start the container attached to the Docker custom network

## Work in Progress Instructions
## Testing

In the current incarnation, when the service starts it creates the sqlite database schema, and populates initial data therein. This includes the admin user, ui_service_admin. A token for this user may be used to add alerts.
After running `kb-sdk test` the first time, which generates `test_local`:

See the scripts ```test/manual```. Running the script 'send.sh' with an argument set to the base name for an accompanying json file will send the associated json-rpc payload.

E.g.
- For test_local/test.cfg set the test_token to a valid login token.
- Generate an ssl self signed certificate and place into test_local/workdir: `test-ssl.crt` and `test-ssl.key`.

```bash
KB_TOKEN=XXXMYTOKENXXX bash send.sh add-alert
```

## Development

### install mongo image

I use Kitematic to get the latest in the 3 series (currently 3.7.9) - the 2 series which kbase uses is not supported in the official distribution.

Change these settings:

- enable kbase-dev network
- map directories to SPRINT/mongo/db and SPRINT/mongo/configdb

### Add ui_service user and database

mongo localhost:PORT

where port is the port reported in Kitematic (it changes with every container startup)
mkdir test_local/workdir
openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout test_local/workdir/test-ssl.key -out test_local/workdir/test-ssl.crt -subj "/C=US/ST=California/L=Berkeley/O=LBNL/OU=KBase/CN=ci.kbase.us"
```
Loading