- Web Scraping for Data Analysis
- Predictive Modelling on Customer's Data
- Conclusion
- Libraries Utilized
Web scraping was employed to gather customer reviews and insights about Air India from the website Airline Quality. Using web scraping techniques, data was extracted from the website, including customer comments, ratings, and other relevant information, which was then compiled into the "Reviews Dataset" for further analysis, such as predicting customer buying behaviors or understanding customer sentiments towards Air India's services.
Data preprocessing is crucial in the data mining process, involving cleaning, transforming, and integrating data for analysis. Its goal is to enhance data quality and suitability for specific tasks
Data cleaning is the process of fixing or removing incorrect, corrupted, incorrectly formatted, duplicate, or incomplete data within a dataset.
- Removal of sentences before '|' in dataframe
- Removal of all special characters from the dataframe
- Tokenization is the process of dividing text into a set of meaningful pieces.
- Tokens converted to tuples using POS Tagging, grouped into words through lemmatization.
Sentiment analysis is the process of analyzing digital text to determine if the emotional tone of the message is positive, negative, or neutral.
- VADER(Valence Aware Dictionary for Sentiment Reasoning) is an NLTK module that provides sentiment scores based on the words used.
- It is a rule-based sentiment analyzer in which the terms are generally labeled as per their semantic orientation as either positive, negative or neutral.
Data visualization uses graphics like charts, plots, infographics, and animations to represent complex data relationships and provide easy-to-understand insights.
Matplotlib is a comprehensive library for creating static, animated, and interactive visualizations in Python.
![](https://private-user-images.githubusercontent.com/88373687/256597268-03e1d670-39bb-4c64-aedd-ccdeb2c39755.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjU2MDIzODAsIm5iZiI6MTcyNTYwMjA4MCwicGF0aCI6Ii84ODM3MzY4Ny8yNTY1OTcyNjgtMDNlMWQ2NzAtMzliYi00YzY0LWFlZGQtY2NkZWIyYzM5NzU1LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MDYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTA2VDA1NTQ0MFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWRhODBlZmRlYzk5YTQyYTMwYjMxOWE3YzQ2MzM4NWI0YzIzNGQwZjM1YzIwMDNiMzU1ZmIyZmU5YmFhZjlkN2YmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.dDgP-8f3qUUBCp57HhDlQqXs11VzUcrKsdU5KXch9zM)
Wordcloud is basically a visualization technique to represent the frequency of words in a text where the size of the word represents its frequency.
![](https://private-user-images.githubusercontent.com/88373687/256597757-d3770498-5ab5-4707-a44d-5482123565ec.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjU2MDIzODAsIm5iZiI6MTcyNTYwMjA4MCwicGF0aCI6Ii84ODM3MzY4Ny8yNTY1OTc3NTctZDM3NzA0OTgtNWFiNS00NzA3LWE0NGQtNTQ4MjEyMzU2NWVjLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MDYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTA2VDA1NTQ0MFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTY4NzY2MzYwNWU1NjZlMDc5NTdkMjFhMGQ3M2UxNzI2MTcwY2Q2NTc3ZmY1NDlhYzE5N2FmMjUwOTkyN2RkZTAmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.0ftoUDOP7DmBtcy2fflTt8PQOWYfVGQLw4GvOq7tLfw)
Predictive models are machine learning algorithms trained on high-quality customer data, requiring manipulation and preparation to accurately predict target outcomes.
- Exploratory Data Analysis is a crucial step in the data analysis process, where the primary goal is to understand the data, gain insights, and identify patterns or relationships between variables.
- Imported Chardet library(Universal Character Encoding Detector) for UTF-8 encoded code, applied to CSV, and checked for null values.
- MI score graphs visualize feature relevance to the target variable, measuring dependency and aiding feature selection.
- The scikit-learn (sklearn) library calculates the MI_score correlation between attributes, and a graph is plotted for visualization purposes.
![](https://private-user-images.githubusercontent.com/102895768/257048647-9206150b-b50a-4197-9957-0d917629a928.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjU2MDIzODAsIm5iZiI6MTcyNTYwMjA4MCwicGF0aCI6Ii8xMDI4OTU3NjgvMjU3MDQ4NjQ3LTkyMDYxNTBiLWI1MGEtNDE5Ny05OTU3LTBkOTE3NjI5YTkyOC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwOTA2JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDkwNlQwNTU0NDBaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1iZTFmOTcwMjVmM2NmNjliZGRjMTc4ZDljZjRiZmYwZWNjZWQyZTBjYmM4Y2ZjYmY5NDJmMmUyNDA0YTIwZTQyJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.iRAPCjdG2qqecSVFRJg1oVfbyz-71V7Jfw1alXJc6YU)
- Test and train split is a crucial step in building and evaluating machine learning models, dividing datasets into training and test sets.
- Training sets contain 70-80% of data, while test sets allocate 20-30%.
- The code splits data into training, validation, and testing sets, ensuring model training, validation, and testing on different subsets, preventing overfitting, and providing a reliable evaluation.
MinMaxScaler
Min-Max Scaling is a preprocessing technique for scaling numerical features to a fixed range, ensuring consistent scaling across all features.
Random Forest is an ensemble learning method combining multiple decision trees, capturing complex relationships and interactions for more accurate and robust models.
- For top-6 features (Accuracy = 74.5762)
- For all features (Accuracy = 71.1864)
XGBoost is a popular machine learning algorithm utilizing gradient boosting to optimize model performance and computational efficiency.
- For top-6 features (Accuracy = 71.1864)
- For all features (Accuracy = 71.1864)
Validating the model on the test dataset is an essential step in the machine learning workflow to assess how well the model performs on unseen data.
- Accuracy = 71.1864
The Random Forest classifier with the top 6 features showed slightly higher accuracy than XGBoost. It can predict customer satisfaction or other target variables in datasets. Performance may vary depending on data quality and representativeness.
- BeautifulSoup (bs4)
- Chardet
- Matplotlib
- Natural Language Toolkit (nltk)
- Numpy (np)
- Pandas (pd)
- Requests (re)
- Seaborn (sns)
- Scikit-learn (sklearn)
- VaderSentiment (SentimentIntensityAnalyzer)
- Warnings
- WordCloud