تحليل تغريدات تويتر

تحليل تغريدات تويتر عن الزحمة في ثلاث مدن وإظهار النتائج باستخدام خوارزميات تعلم الأله SVM

عمل بحث أكاديمي بناء على البرمجة بما لا يقل عن 7000 كلمة

Request details

Analyzing Twitter Posts to Support the Development of Smart Cities

Build a system to decide wither there is a traffic issue in a specific area throughout the processing of collective tweets in three cities by using SVM algorithm.

The proposed architecture for the system that help decision maker to make an informative decision about the congestion statue in a specific area based on sentiment analysis of tweet consist of five modules:

Data collection module

Data preprocessing module

Data classification module using Support Vector Machine (SVM)

Data aggregation module

Data visualization module

In data collection module, data is going to be collated from Twitter using Twitter Streaming APIs which push tweet objects to end user in near real time. However, there are two filters that are going to be applied to the data coming from Twitter stream;

•Filter 1: Tweet objects attributes that should be collected are text attribute and geolocation attribute. Other tweet attributes are not relevant.

•Filter 2: Tweet text attribute which contains the two keywords; “traffic” and “congestion” and their synonyms.

Twitter allows end user to specify the area of collection. That’s it, it uses the concept of bounding-box to specify the area of collection of tweets. Bounding-box is a rectangular area obtained with two coordinates pairs; latitude and longitude.

In data preprocessing module, text processing takes place to further prepare the text within a tweet for classification. This is required to solve a range of informality in the text such as the existence of slags, jargons, abbreviations, pictures, and punctuations signs. Text preprocessing involves:

•Lowercasing – for performance issue, it is important to unify the text so that the system dese not process the same word two times. For example, “Liverpool” and “liverpool” are similar in term of representation and interpretation, however, without lowercasing them, they are going to be processed as two different words.

•Tokenization – this involves dividing a sentence into a list of words. Each word is referred to as token.

•Repeated characters transformation – this involves transforming a sequence of repeated characters. For instance, “E” is transformed into “Errr”.

•Clearing – this involves removing URLs, hashtags, and digits from the tweet text. This also covers removing stop and short words.

In data classification module, a training dataset is required. Since there is no gold standard dataset in this domain, there is a need to create training and testing dataset. Testing dataset is very important for testing and evaluation the performance of classification algorithm.

Classification requires a group of features that represent an object. By comparing (i.e. computing) distance between the object group of features, an object class is determined, and the SVM is used for this purpose. That it, SVM is going to be trained to extract the features of each class so that comparison or classification can take place on testing dataset.

The commonly used methods for characterizing the text are Bag-of-Words (BoW) and Bag-of-Embeddings (BoE). The former is a matrix where each entry is associated to a sentence and takes the form of term-frequency vector. The latter is based on learning distributed representation of words. Each word is represented by a distribution of weights. The algorithms for BoE and BoW are available in Python library called “scikit learn” and “gensim”.

In aggregation module, the correctly classified tweets as “congestion” are going to be filtered based on location attribute into one group and store them for visualization stage.

In visualization module, data is going to be presented in visual form for decision maker to see the distribution of congestion on map. Python library (Plotly) is going to be used in plotting data on map.

Expected Outcome

This system is created to help decision maker to make an informed decision whether an area in a city is congested or not based on people tweets. Thus, the output is binary:

•positive : indicates congestion

•negative: indicates no congestion

At the level of classification algorithm, the expected output is based on tweet instance. A tweet can be said:

•positive; indicates that the sentiment in the tweet acknowledges the existence of congestion in an area. An example of such tweet is “Some traffic congestion in New York city center as the teams arrive home”.

•negative: indicates that the sentiment in the tweet acknowledges the non-existence of congestion in an area, yet the tweet sentiment still contains traffic description words. An example of such tweet is “The gov has plan to reduce traffic congestion around New York city center”.

Doing a research about this project and describing how the code works, what the steps are followed and why these steps are used rather than others with at least 7000 words using visualization outcomes.