Database for Datascientist? Easy and quick solution with MongoDB and Pandas+Streamlit (Part 2)

Introduction

In the previous article, we have seen how to create a database with MongoDB and how to use it with Pandas. In this article, we will see how to use MongoDB with Streamlit and deploy as a web application.

Streamlit

Streamlit is an open-source Python library that makes it easy to create and share beautiful, custom web apps for machine learning and data science. It is a great tool for data scientists and machine learning engineers to create web applications for their projects. It is very easy to use and has a lot of features. It is also very easy to deploy on Heroku.

The nice thing about Streamlit is that it is very easy to use. You can create a web application with just a few lines of code, no HTML, CSS, or JavaScript required.

And the other nice thing is that you can deploy for free on different platforms like Heroku, AWS, and Google Cloud.

Installation

To install Streamlit, is very easy, it can be done with pip in your dedicated python environment.

pip install streamlit

After installing Streamlit, to verify that it is installed correctly, you can run the following command in your terminal.

streamlit hello

This will open a new tab in your browser with a simple web application as shown in the image below.

Create a basic web application with Streamlit and plug it to MongoDB

In the following section we will create a basic web application with Streamlit that retrieves data from MongoDB and displays it in a table. The app is very basic and it is just to show how to use Streamlit with MongoDB, so don't expect a lot of features, it will only display the data in a table built with st.dataframe() :-). If you are interested in more advanced features of Streamlit, write a comment below and I will write another article about it :-).

The dataset: Seattle Airbnb Listings

The dataset I used for this example is the Seattle Airbnb Listings dataset. It contains information about the listings of Airbnb in Seattle. The dataset is available on Kaggle.

I will not explain in details here how I loaded on my MongoDB database, but for this exercise you can upload the csv file using the MongoDB Compass GUI, you can find the documentation here. It is quite easy to do. You can also do it programmatically, using the pymongo Python API for MongoDB. See below the screenshots of how easy is to add a csv file into your database using MongoDB Compass.

1. Add the dataset to MongoDB Compass

2. Import CSV and JSON files into MongoDB Compass

3. Dataset visualization as collection in MongoDB Compass

The code: let's build our simple web application

Now that we have our dataset in MongoDB, we can start building our web application. It is a python file so we need to save it with the .py extension. I called it app_st.py(not very creative, I know :-)).

1. Import the libraries

import pandas as pd
import numpy as np
import pymongo
from pymongo import MongoClient
import streamlit as st

2. Connect to MongoDB

After importing the libraries, we need to connect to our MongoDB database. We can do it using the MongoClient() function from the pymongo library. We need to specify the host and the port of our MongoDB database. In my case, I am using the default host and port, so I don't need to specify them. I masked my user name and password of course :-).

## connect to mongodb
client = MongoClient('mongodb+srv://User:[email protected]/?retryWrites=true&w=majority')

3. Get the database and the collection

After connecting to MongoDB, we need to specify the database and the collection we want to use. In my case, I am using the database "Marcello_test"(my name...) and the collection "Seattle_listing".

#  Get an existing database named "Marcello_test"
db1 = client.Marcello_test

#   Get a collection named "Seattle_listing"
collection_main = db1.Seattle_listing

4. Parsing the documents from MongoDB to Pandas

Now that we have the collection, we can parse the documents to Pandas. We can do it using the find() function from the pymongo library. We can also specify the fields we want to retrieve from the collection. In my case, I am retrieving all the fields.

# save the documents in a dataframe
df = pd.DataFrame(list(collection_main.find()))

df1 = df.drop(['_id'], axis=1) # drop the _id field, not needed, it is created automatically by MongoDB

5. Create the web application with Streamlit

Now that we have the data in a Pandas dataframe, we can create our web application with Streamlit. We can use the st.dataframe() function to display the data in a table. We can also use the st.title() function to add a title to our web application.

# setting the screen size

st.set_page_config(layout="wide",
                   page_title="Seattle Airbnb Data")
                  

# main title
st.title('Seattle Airbnb Listings MongoDB') # title of the web application

# main text
st.subheader('This app is a Streamlit app that retrieve mongodb data and show it in a dataframe') # subheader

st.write('Data: Sample of Seattle listing with Name, Description, Price') # write text and description

st.dataframe(df1) # display the dataframe in a table

All done!! with just a few lines of code we have created a web application that retrieves data from MongoDB and displays it in a table.

6. Run the web application locally with Streamlit

Now that the code is ready, we can run the web application locally with Streamlit. We can do it using the following command in the terminal.

streamlit run app_st.py

This code will open a new tab in your browser with the web application as shown in the image below, using a local host.

Et voi la! We have created a web application that retrieves data from MongoDB and displays it in a table.

FilmForth_tutorial_streamlit_mongodb.mp4