Most of the Phase 3 was spent on creating the DataSet library, tweaking it, and creating blog posts.

A list of issues and PRs created uptil Phase 2 are:

Here is the list of blog posts describing the contributions to Pharo:

1. Summary of the proposal
2. Community bonding period
3. Phase 1 progress
4. Phase 2 progress
5. Phase 3 blogs: DataSet library, DataFrame IO comparison and DataFrame Joins.

You can view all the blog posts at this link.

Future work

Improving DataFrameTypeDetector

As described in a previous blog post, improving the DataFrameTypeDetector class will lead to faster load times of csv files.

DataSet library

Identifing and adding additional datasets to the library. Also, modifing the library such that adding new datasets would require adding minimal amount of code.

Tutorials

Creating data analysis tutorials using the new DataSet library, DataFrame and PolyMath together.

Additional improvements to DataFrame

Refer to the roadmap of the DataFrame library. Features like database IO, time series, spec editor etc need to be added to the library.

1. Boston house prices dataset
2. Breast cancer wisconsin (diagnostic) dataset
3. Diabetes dataset
4. Optical recognition of handwritten digits dataset
5. Iris plants dataset
6. MNIST testing data
7. Wine recognition dataset

Any requests for additional datasets to be added can be put at this issue.

Structure and Working of the library

The DataSetFiles repo has datasets in csv format, and the DataSet library has the code to fetch and convert it as a DataFrame object.

To load a dataset, you use DataSet loadXYZ. For example:

1

df := DataSet loadBoston.

This checks the filesystem to see if the dataset is available, and downloads if it isn’t present using DataSet downloadXYZ method. The downloaded files are stored at data folder in the root of repo stored in the filesystem.

To download all the datasets in advance, use:

1

DataSet downloadAll.

This downloads the datasets that have not yet been downloaded.

Future work

A possible improvement to the library would be storing a list of datasets in DataSetFiles, and modifing DataSet library such that user will be able to dynamically fetch available datasets and download whichever is needed. If you have thoughts/suggestions regarding this or any improvements, email me or create an issue on Github!

As you can see, the fastest method is json with orient='split', since the format is close to the object’s internal representation. Next fastset is the csv and json orient='records'. Others are significantly slower.

The primary reason for slowdown in reading csv is the DataFrameTypeDetector detectTypesAndConvert method. It runs through all elements in all columns and determines the type of that column. It may run multiple times on a column, causing a significant slowdown for large dataframes. On large datasets, it has slowdown of at least 85%. Modifying this method to query on only a few elements will cause a significant performance boost.

Similarly, for reading json using the fastest method available (orient: split), detectTypesAndConvert method causes slowdown.

TL;DR: Improving DataFrameTypeDetector will massively increase reading speeds, by 50% to 90% for large files.

Joining DataFrames

The library supports SQL-type joins: inner, outer, left, right.

Joining using rowNames

Joining dataframes using rowNames is faster vs an arbitary column, since rowNames are unique leading to faster construction of output dataframe.

Consider two dataframes with course names as rowName:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

df1 := DataFrame withRows: #(
	('RTH-01', '11:00 - 01:00')
	('RTH-02', '04:00 - 05:50')
	('THH-02', '09:00 - 10:30')
	)
	rowNames: #('CS-10' 'CS-11' 'CS-13')
	columnNames: #(Location Timing).

df2 := DataFrame withRows: #(
	('Prof. Foo' 50)
	('Prof. Bar' 45)
	)
	rowNames: #('CS-11' 'CS-12')
	columnNames: #(Instructor Capacity).

df1

	Location	Timing
CS-10	RTH-01	11:00 - 01:00
CS-11	RTH-02	04:00 - 05:50
CS-13	THH-02	09:00 - 10:30

Inner join combines two dataframes based on their intersection (in this case, the rowNames). Doing df1 innerJoin: df2 would output intersecting rows between dataframes (CS-11) with columns combined.

Outer join includes all the rows in the dataframe, and adds nil to the missing columns.

Left join includes all the rows present in the left dataframe, while adding nil in places with missing data. CS-10, CS-11 and CS13 will be present in the output of df1 leftJoin: df2, with nil in column Instructor and Capacity for rows CS-10 and CS-13.

df1 leftJoin: df2

	Location	Timing	Instructor	Capacity
CS-10	RTH-01	11:00 - 01:00	nil	nil
CS-11	RTH-02	04:00 - 05:50	Prof. Foo	50
CS-13	THH-02	09:00 - 10:30	nil	nil

Right join is similar to left join, except it preserves the rows in the right dataframe.

df1 rightJoin: df2

	Location	Timing	Instructor	Capacity
CS-11	RTH-02	04:00 - 05:50	Prof. Foo	50
CS-12	nil	nil	Prof. Bar	45

Joining using arbitary column

The API for joining arbitary column is:

1
2
3
4

df innerJoin: df2 onLeft: 'LeftCol' onRight: 'RightCol'.
df outerJoin: df2 onLeft: 'LeftCol' onRight: 'RightCol'.
df leftJoin: df2 onLeft: 'LeftCol' onRight: 'RightCol'.
df rightJoin: df2 onLeft: 'LeftCol' onRight: 'RightCol'.

Consider similar dataframes as before, except with course names being it’s own column:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

df1 := DataFrame withRows: #(
	('CS-10' 'RTH-01', '11:00 - 01:00')
	('CS-11' 'RTH-02', '04:00 - 05:50')
	('CS-13' 'THH-02', '09:00 - 10:30')
	)
	columnNames: #(Course Location Timing).

df2 := DataFrame withRows: #(
	('CS-11' 'Prof. Foo' 50)
	('CS-12' 'Prof. Bar' 45)
	)
	columnNames: #(Course Instructor Capacity).

Joining using left join would lead to:

df1 leftJoin: df2 onLeft: 'Course' onRight: 'CourseName'

	Course	Location	Timing	CourseName	Instructor	Capacity
1	CS-10	RTH-01	11:00 - 01:00	nil	nil	nil
2	CS-11	RTH-02	04:00 - 05:50	CS-11	Prof. Foo	50
3	CS-13	THH-02	09:00 - 10:30	nil	nil	nil

Joining datafames with same column names

If the two dataframes have conflicting column names, it will be replaced by _x and _y. For instance, in the previous example, if both dataframes has column names Course, output dataframe would be:

df1 leftJoin: df2 onLeft: 'Course' onRight: 'Course'

	Course_x	Location	Timing	Course_y	Instructor	Capacity
1	CS-10	RTH-01	11:00 - 01:00	nil	nil	nil
2	CS-11	RTH-02	04:00 - 05:50	CS-11	Prof. Foo	50
3	CS-13	THH-02	09:00 - 10:30	nil	nil	nil

Appending dataframes

Consider two dataframes:

You can append/concatenate them by:

1

appenDf := df1, df2.

There are a few caveats:

1. Row names should be unique. Else, the concatenation operator will not work. You can use df1 rowNames: (1 to: df1 size) and df2 rowNames: (df1 size + 1 to: df1 size + df2 size) followed by df1, df2.
2. Order of columns should be the same. If it is not, reconstruct the dataframe.

DataFrame library

PR#111 and other PRs (102, 103, 104 and 107) have been modified and merged. This means that all issues related to missing data have been fixed! 🎉

In addition to this, two major features have been added - Joins support and JSON I/O.

DataFrame joins

SQL-type joins are now possible between DataFrames. There are two APIs for this. For joins using rowNames, following is used:

1
2
3
4

df innerJoin: df2.
df outerJoin: df2.
df leftJoin: df2.
df rightJoin: df2.

And for joins using arbitary columns, the API is:

1
2
3
4

df innerJoin: df2 onLeft: 'LeftCol' onRight: 'RightCol'.
df outerJoin: df2 onLeft: 'LeftCol' onRight: 'RightCol'.
df leftJoin: df2 onLeft: 'LeftCol' onRight: 'RightCol'.
df rightJoin: df2 onLeft: 'LeftCol' onRight: 'RightCol'.

Joining using rowNames (first API) is faster than using arbitary columns.

JSON I/O

The library can perform can read or write DataFrames from/to .json files. Under the hood, it uses the NeoJSON library to read/write json, and then converts it to DataFrames.

There are five different formats of JSON to which read/write is supported:

1. columns in which the format is of {column:{rowName:data, ...}, ...}
2. index which has the format {rowName:{column:data, ...}, ...}
3. values which is an array of arrays (rows)
4. split in which the format is {columns:[columnNames], index:[rowNames], data:[data]}
5. records which is an array of rows [{col1: data1, ...}, ...]

Out of these, split is the fastest format to read/write into, since it stores DataFrame similar to it’s internal representation.

There are three ways to write to JSON:

df writeTo: aFileRef using: (DataFrameJsonWriter new).

DataFrameJsonWriter new write: df to: aFileRef.

DataFrameJsonWriter new writeAsString: df.

Similarly, you can read JSON to DataFrame object as:

df := DataFrame readFrom: aFileRef using: (DataFrameJsonReader new).

df := DataFrameJsonReader new readFrom: aFileRef.

PolyMath library

I spent most of my community bonding period as well as first week of phase 1 on PolyMath library, exploring codebase, fixing bugs and adding features. My most favorite part was adding t-SNE implementation! Here is a peek at a couple of visualizations:

The visualization is made possible using Roassal3, be sure to check it out at https://github.com/ObjectProfile/Roassal3.

Here is the list of issues I created:

1. Math-TSNE is incomplete
2. PMVector > < operators modify in-place
3. PMVector sum is extremely slow
4. PMStandardScalar fails when scale = 0

And here is the list of PRs that solved them:

1. Implementing the t-SNE algorithm
2. Fix PMVector comparison operators
3. Removed PMVector sum for speedup
4. Removed == method from PMVector
5. Refactored PMTSNE to include steps
6. Added vizualization examples for PMTSNE
7. Fixed ZeroDivideError in PMStandardizationScaler

DataFrame library

The main focus for phase 1 was getting the library work with missing data - initializing with missing values, reading files, and providing methods to fill the data.

Here is the list of issues created by me:

1. Handling missing data - collection of all issues related to missing data
2. DataFrame select: fails when no rows are selected
3. DataSeries does not support boolean operators with scalars
4. Add JSON read/write support
5. DataFrameInternal - Using OrderedCollection over Array2D
6. DataFrame addRow does not consider key order

PRs which solve some of these issues:

1. Boolean operators for DataSeries
2. Added support for DataFrame init with missing values
3. Added ability to remove nil from DataFrame and Series
4. Added DataSeries fillNilsWith method
5. Added method to convert missing values from files
6. DataFrameTypeDetector now works with nil

Next steps

Some of the features planned for phase 2 include implenting joins, adding json support and a new DataSet library. I will be creating a detailed post after the first evaluation (first week of July). Be sure to star and follow DataFrame and PolyMath on Github for updates on the progress! :)

1. t-SNE implementation in Pharo

t-SNE is a visualization algorithm, through which you can judge your input dataset, reveal it’s clusters, etc. The algorithm implemented was incomplete, as mentioned in Issue#115. I worked torwards completing the algorithm, having a running version in the commit 7f3a812. We started using Github Boards for tracking the progress, which can be found here. Overall, I spent about 2 weeks learning the details and implementing the algorithm. I’ll be creating a seperate post detailing the algorithm soon.

2. Speeding up PMVector sum

While profiling t-SNE, I discovered that current implementation of PMVector sum was very slow. sum is used to calculate the sum of PMVector, and is also used in PMMatrix to calculate sums of rows and columns. You can read more about the issue here: Issue#125. It was easily fixed by relying on parent Collection’s sum, which increased speed quite a lot.

3. PMVector comparision operators

PMVector had a redundant and perhaps, wrong operator ==, which was removed (Issue#90, PR#108). Also, operators > and < modified the source vector in-place, causing unintuitive behavior while working with PMMatrix: Issue#122, PR#123.

First two weeks were mostly spent on learning about PolyMath and it’s classes, integrating Roassal and DataFrame with PolyMath, and exploring codebase for DataFrame. Overall, it has been productive 3 weeks!

To demonstrate this, here is a piece of code utilising all three libraries:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82

"Iris plotting in Roassal"
reader := DataFrameCsvReader new.
fileRef :=  'iris.csv' asFileReference.
df := DataFrame readFrom: fileRef using: reader.

df_x := df columns: #('Sepal length' 'Sepal width' 'Petal length' 'Petal width').

dataServer := PMMemoryBasedDataServer new.
dataServer data: df_x.

finder := PMClusterFinder new: 3 server: dataServer type: PMEuclideanCluster.
finder minimumRelativeClusterSize: 0.01.
clusters := finder evaluate.

y := DataSeries new name: 'Output'.
df withIndexDo: [ :row :index |
	y add: index->(finder indexOfNearestCluster: ({row at: 'Sepal length'. row at: 'Sepal width'. row at: 'Petal length'. row at: 'Petal width'} asPMVector)).
	].

df addColumn: y.

m := PMMatrix rows: df_x.
m := (PMStandardizationScaler new) fitAndTransform: m.
pca := PMPrincipalComponentAnalyserJacobiTransformation new componentsNumber: 2.
pca fit: m.
reduced := pca transform: m.

transformOutput := Dictionary newFrom: {
	3->'Iris-setosa' .
	2->'Iris-versicolor' .
	1->'Iris-virginica' .
}.
df column: 'Output' transform: [ :column |
	column collect: [ :number |
		transformOutput at: number.
	].
].

df addColumn: (reduced atColumn: 1) named: 'PCA x'.
df addColumn: (reduced atColumn: 2) named: 'PCA y'.

b := RTGrapher new.

ds_setosa := RTData new.
ds_setosa label: 'Iris setosa'.
ds_setosa dotShape circle color: Color red trans.
ds_setosa points: (df select: [:row | ((row at: #Type) = 'Iris-setosa') & ((row at: #Type) = (row at: #Output))]).
ds_setosa interaction popupText: [ :row | 'Actual: ', (row at: #Type) asString, '. Predicted: ', (row at: #Output) asString ].
ds_setosa x: [ :row | row at: 'PCA x' ].
ds_setosa y: [ :row | row at: 'PCA y' ].
b add: ds_setosa.


ds_versicolor := RTData new.
ds_versicolor label: 'Iris versicolor'.
ds_versicolor dotShape circle color: Color blue trans.
ds_versicolor points: (df select: [:row | ((row at: #Type) = 'Iris-versicolor') & ((row at: #Type) = (row at: #Output))]).
ds_versicolor interaction popupText: [ :row | 'Actual: ', (row at: #Type) asString, '. Predicted: ', (row at: #Output) asString ].
ds_versicolor x: [ :row | row at: 'PCA x' ].
ds_versicolor y: [ :row | row at: 'PCA y' ].
b add: ds_versicolor.


ds_virginica := RTData new.
ds_virginica label: 'Iris virginica'.
ds_virginica dotShape circle color: Color green trans.
ds_virginica points: (df select: [:row | ((row at: #Type)) = 'Iris-virginica' & ((row at: #Type) = (row at: #Output))]).
ds_virginica interaction popupText: [ :row | 'Actual: ', (row at: #Type) asString, '. Predicted: ', (row at: #Output) asString ].
ds_virginica x: [ :row | row at: 'PCA x' ].
ds_virginica y: [ :row | row at: 'PCA y' ].
b add: ds_virginica.

ds_misclassified := RTData new.
ds_misclassified label: 'Misclassified'.
ds_misclassified dotShape circle color: Color black trans.
ds_misclassified points: (df select: [:row | (row at: #Type) ~= (row at: #Output)]).
ds_misclassified interaction popupText: [ :row | 'Actual: ', (row at: #Type) asString, '. Predicted: ', (row at: #Output) asString ].
ds_misclassified x: [ :row | row at: 'PCA x' ].
ds_misclassified y: [ :row | row at: 'PCA y' ].
b add: ds_misclassified.

b.

Here is it exported from Roassal as HTML (go on - hover over the points!):

In the last week, I begun implementing t-SNE for PolyMath. It is exciting to dissect the paper, working through the nitty-gritty details and translating them into code. By the end of next week, it will be the first paper I have implemented! I also am writing an accompanying post, which will explain the math behind the algorithm - it is one of the reason that this week’s post is small!

You can track the progress of the implementation at this link: t-SNE project board.

About Pharo

Pharo is an easy-to-learn, pure object oriented programming language based on smalltalk. Pharo comes with it’s integrated IDE, which makes browsing code, documents and examples easier, as well as it’s inspector and inbuilt debugging tool allows you for faster development. It also has various libraries like Seaside (web-dev), PolyMath (scientific computing), Roassal (Visualization) etc. Overall, it is a great language for learning object oriented programming as well as enterprise applications.

About DataFrame library

The dataFrame library was first created as a GSoC 2017 project by Oleksandr, which introduced multiple functionalities making it fit for data analysis. The DataFrame Booklet consists of documentation regarding current API. I have used this library as a base for the Whatsapp Analyzer project.

The Plan

1. Handling missing data

Advanced missing data functionality such as detecting different types of missing data (NA, nill, ?), replacing them, reading files with incomplete data etc. will be added. This should also enable detecting column-type functionality of dataframes.

2. Joins between dataframe

Merging dataframes using joins (left, right, inner) is crucial and will be supported by end of summer

3. JSON import/export

Json is frequently used for transmitting data over the network, and having a json import/export functionality will enable creating and consuming dataset API endpoints over the internet.

4. Mathematical operations

Different operations on DataFrame and DataSeries are planned, such as correlation, covariance, cummulative (min, max, product, sum), clip, and different operators such as >, <, >=, <=, mod, pow. This will bring flexiblity to the library.

5. Dataset fetcher

A new Dataset fetcher would also be added to fetch popular datasets such as Iris, Boston, Mnist etc., making it easier to experiment with the library.

6. Documentation and tests

A good amount of documentation would be added ranging from examples of the API to the comments of the messages, along with additions of tests. The aim is to make the library easy to use for the user

Detailed proposal

The detail proposal has been uploaded here, which has the timelines and messages that will be implemented.

Next steps

The community bonding period is till May 27th, in which I am going to explore the PolyMath library and fix some issues, as well as refine the tasks and schedule for this summer.

You can track the progress of this project by clicking on the tag “GSoC progress” at below, or view the monthly posts by clicking on “GSoC summary”.

Prerequisites

1. Parsed df object

Follow Part 1 to obtain a df object with messages stored in it. A sample chat has been added for convenience.

2. ChatFeatures and ChatAnayzer classes

These classes defined in Part 2 will be used in this part. You can also clone the repo mentioned at the end of part 2 to obtain them.

Preparation

Let’s create the following class, which will be used to analyze the dataframe.

1
2
3
4

Object subclass: #ChatAnalyzer
   instanceVariableNames: ''
   classVariableNames: ''
   package: 'WhatsappAnalyzer'

We’ll also create a class-side method:

1
2
3
4

getAuthorsFrom: df
   "Returns an OrderedCollection of authors present in the dataframe"

   ^ (df column: #Author) asSet.

Analysis

Basic Analysis

The total messages exchanged can be easily found out by df size, since we have already removed system messages (eg - ABC changed the group name to XYZ) and handled multi-line messages in part 2. We can infer messages like: This message was deleted as well as Media <Media Omitted> by using reject similar to ChatCleaner removeMessages: from. (Note: The error block is due to a bug in the library, which will be fixed soon)

We will store all results in a Dictionary, which can be later used to turn into a json format using NeoJSON.

Here is a basic method which analyzes basic message exchanges:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

getMessageCounts: df
   "Counts total messages, text messages, media and deleted messages"

   | messageCounts tempDf |
   messageCounts := Dictionary new.
   messageCounts add: 'Total'->(df size).
   [
      tempDf := df select: [ :row | (row at: #Message) = '<Media omitted>' ].
      messageCounts add: 'Media'->(tempDf size).
   ]   ifError: [ 
      messageCounts add: 'Media'->0.
       ].
   [
      tempDf := df select: [ :row | (row at: #Message) = 'This message was deleted' ].
      messageCounts add: 'Deleted'->(tempDf size).
   ]   ifError: [ 
      messageCounts add: 'Deleted'->0.
       ].
   messageCounts add: 'Text'->((df size) - (messageCounts at: 'Media') - (messageCounts at: 'Deleted')).
   ^ messageCounts

You can use it as:

1

messageCounts := ChatAnalyzer getMessageCounts: df.

By filtering Author field, we can do same for getting each author’s message counts.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

getAuthorMessageCounts: df
   "Generates message counts for each author"

   | messageCounts authors authorDf |
   messageCounts := Dictionary new.
   authors := (df column: #Author) asSet.
   authors do: [ :author |
      authorDf := df select: [ :row | (row at: #Author) = author].
      messageCounts add: author->(self getMessageCounts: (authorDf)).
       ].
   ^ messageCounts 

You can add it to previous messageCounts as:

1

messageCounts add: 'Authors'->(ChatAnalyzer getAuthorMessageCounts: df).

Now, we can extract metrics such as most active user, average message per user and others. To group them, we create yet another method:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

getBasicTextAnalysis: df
   "Returns a dictionary with following metrics:
      Most active user w/count
      Most active media user w/count
      Messages/user 
   "

   | messageCounts textAnalysis|
   textAnalysis := Dictionary new.
   messageCounts := self getMessageCounts: df.
   messageCounts add: 'Authors'->(self getAuthorMessageCounts: df).
   
   "Calculate most active user"
   textAnalysis add: 'Most active user'->''.
   textAnalysis add: 'Most active user messages'->0.
   
   "Average messages per user"   
   textAnalysis add: 'Messages/user'->0.  
   (messageCounts at: #Authors) keysAndValuesDo: [ :key :value |
      ((textAnalysis at: 'Most active user messages') < (value at: 'Text')) ifTrue: [ 
            textAnalysis at: 'Most active user messages' put: (value at: 'Text').
            textAnalysis at: 'Most active user' put: key.
          ].
      textAnalysis at: 'Messages/user' put: ((textAnalysis at: 'Messages/user') + (value at: 'Text')).
       ].
   
   textAnalysis at: 'Messages/user' put: (textAnalysis at: 'Messages/user') asFloat / ((messageCounts at: #Authors) size).
   messageCounts add: 'Basic Text Analysis'->messageCounts.
   
   ^ messageCounts

We iterate through summary of each Author’s messages using messageCounts at: 'Authors' keysAndValuesDo: and count occurences as needed. For example - Most active user is found out by finding max of Text key for each Author.

Additional fields for this analysis is present on my repo. Here is how the resulting messageCounts will look:

Emojis

1. Most frequently used emojis

In the last part, we added ChatCleaner getEmojisFrom: which returned a dataseries with only emojis present. We’ll iterate through the series counting the emojis encountered and storing it in emojiCount which is a DataSeries.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

getEmojiCountFrom: emojiDs
   "Returns an DataSeries having an emoji as key and counts as values"

   | emojiCount |
   emojiCount := DataSeries new name: #EmojiCount.
   emojiDs do: [ :message |
      message do: [ :emoji |
         emojiCount at: emoji
            transform: [ :count | count + 1 ]
            ifAbsent: [ emojiCount add: emoji->1 ].
         ]
      ].
   ^ emojiCount sortDescending.

We can now get this result:

1
2
3

emojiDs := ChatCleaner getEmojisFrom: df.
ChatAnalyzer getEmojiCountFrom: emojiDs.
a DataSeries($🏻->7 $👍->5 $♂->3 $‍->3 $🙋->3 $🙂->2 $😅->1 $🤷->1 $🤔->1 $🎉->1 $😂->1 $🤨->1 $😁->1 $😓->1)

1
2
3

ignoreEmoji := Set withAll: #(127995 9794 8205 2640).
"and add following in main body before line 8"
(ignoreEmoji includes: (emoji asUnicode)) ifFalse:

1

 a DataSeries($👍->5 $🙋->3 $🙂->2 $😅->1 $🤷->1 $🤔->1 $🎉->1 $😂->1 $🤨->1 $😁->1 $😓->1)

2. Emojis used per person

We can extend our previous method by filtering each author and counting emojis used by that author.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

getMemberEmojiCountFrom: df
   "Returns a dictionory of DataSeries having emoji count of member"

   | emojiPerPerson authors authorDf authorEmojiCount |
   emojiPerPerson := Dictionary new.
   authors := (df column: #Author) uniqueValues.
   authors do: [ :author |
      authorDf := df select: [ :row | (row at: #Author) = author].
      authorEmojiCount := self getEmojiCountFrom: (ChatCleaner getEmojisFrom: authorDf).
      (authorEmojiCount isEmpty) ifFalse: [
         emojiPerPerson add: author->authorEmojiCount.
         ]
       ].
   ^ emojiPerPerson

We filter an author’s posts in line 8, get emoji count of resulting dataframe, and add it into emojiPerPerson. Here is how the output looks like for the sample chat provided in part 1: Inspector is unable to render some unicode characters, however you can export it to csv and see the output.

Frequently used phrases

To see phrases commonly used in a chat, we can count ngrams across all messages. This is done by iterating through the messages and adding ngrams in a dictionary with key as ngram and count as value, similar to counting of emojis.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

getNgramCount: ngram columnName: columnName
   "Returns a DataSeries with ngram count."

   | nGramCount |
   nGramCount := DataSeries new name: columnName.

   ngrams do: [ :ngramCollection |
     ngramCollection do: [ :ngram |
       nGramCount at: ngram
               transform: [ :count | count + 1 ]
               ifAbsent: [ nGramCount add: ngram->1 ].
        ].
      ].

   ^ nGramCount sortDescending.

You can run it and see the output using:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

blacklist := Set withAll: #('This message was deleted' '<Media omitted>').

nGramDf := df deepCopy.
nGramDf := ChatCleaner removeMessages: blacklist from: nGramDf.
nGramDf column: #Message put: ((ChatCleaner getWordsFrom: nGramDf) asArray).
nGramDf column: #Message put: ((ChatCleaner messagesAsLowercase: nGramDf) asArray).
nGramDf column: #Message put: ((ChatCleaner removeStopwordsFrom: nGramDf) asArray).
nGramDf addColumn: (ChatFeatures getNgramsFromDataFrame: nGramDf withN: 3) named: #Ngrams.

nGramCount := ChatAnalyzer getNgramCount: (nGramDf column:#Ngrams) columnName: #NgramCount.

To summarize, line 2 removes all punctuation and emojis, line 3 converts remaining characters to lowercase, and line 4 strips the messages of the stopwords. Line 5 adds a new column with ngrams which is used to create nGramCount.

Here is how the output looks like: The left is with stopwords removed and right is without. Removing stopwords is only effective if the messages are bigger. Also, the ngram methods in ChatCleaner accept ngrams less than n to prevent null strings. It can be modified to having only n-length ngrams by making sure buffer is full before appending to array.

The above method can be applied to per-author by applying filters, as done in basic analysis.

DateTime analysis

Grouping messages by date or time can allow us to find most frequent hour/date of the chat history. In Pharo, you can group objects using the method group: by: aggregateUsing:, such as for finding most frequent date, we can:

1
2
3
4

df
   group: #Message
   by: #Date
   aggregateUsing: #size.

It first groups messages by date, and calculates the size of the created group, returning a DataSeries.

As for the most frequent hour, we need to first convert time strings into hour, followed by grouping.

1
2
3
4
5
6
7
8
9

df addColumn: ((df column: #Time) collect: [
      :element |
      element asTime hour
   ]) named: #Hour.

df
   group: #Message
   by: #Hour
   aggregateUsing: #size.

We can do the same for each user by filtering Author. Here is how it will look for Member26:

1
2
3
4
5

(nGramDf select: [ :row | (row at: #Author) = 'Member26' ])
   group: #Message
   by: #Hour
   aggregateUsing: #size.
>>> a DataSeries(9->20 12->15 13->1)   

This indicates that user is active mostly around 9am and 12pm.

Visualization

Following is the analysis of one of my chats. Visualisation is done using Roassal. A good resource to learn it is http://agilevisualization.com/, and from it’s inbuilt browser containing multiple examples. I did not get enough time to learn Roassal completely so my scripts are bit of a hack, that’s why I have just linked the output. If you are curious about the source code, or wish to see a detailed part on this, mail me.

1
2
3
4
5
6
7

Totals:
   Letters exchanged: 413,040
   Words exchanged: 80,474
   Media: 285
   Deleted: 132
   Total messages: 20,584
   Duration: 2017-02-10 to 2019-04-13

1
2
3
4
5
6
7

Basic text analysis:
   Most active user: Member1
   Most active user messages: 12104
   Most active user percentage: 60.02%
   Most active media user: Member1
   Most active user media: 226
   Average messages/user: 10,083.5

Message distribution by user:

1

Most common emoji: 😂 appeared 2344 times (80% of all emojis).

Top 10 emoji-wheel for member 1:

1

😂, 🤣, 🤦, 😛, 😅, 😒, 🤭, 😌, 😏, 🔥

Messages per hour by members:

Here is messages exchanged vs month:

NameCloud for n=1:

NameCloud for n=3:

Next steps

With the last 3 tutorials, you’ll have tools to analyze your own chat data. You can extend this to other chats such as discord or even mailing list, or add additional insights such as:

• member which initiates conversations
• member’s behavioral pattern
• trending topics across chats/months

and so on.

If you spot any error in this series or can think of additional analysis that can be done, feel free to email me!

Prerequisites

1. WhatsappReader class

The WhatsappReader class provides way to parse exported chat into the dataframe. Follow steps of the last part to create such class.

2. A parsed dataframe

You must also have a df object containing parsed messages. You can create one using last code snippet in previous tutorial.

Preparation

We will create two classes - ChatCleaner and ChatFeatures.

ChatCleaner would contain messages to clean the chat, such as converting to lowercase, removing stopwords etc.

1
2
3
4

Object subclass: #ChatCleaner
  instanceVariableNames: ''
  classVariableNames: ''
  package: 'WhatsappAnalyzer'

ChatFeatures would provide messages to extract features from the chats, such as ngrams.

1
2
3
4

Object subclass: #ChatFeatures
  instanceVariableNames: ''
  classVariableNames: ''
  package: 'WhatsappAnalyzer'

Generating n-grams

n-grams are sets of n consecutive words/characters extracted from a string. These are generally used by aggregating n-grams and extracting top by count. eg: n-grams (n=2) of the string I am feeling lucky! will be I am, am feeling, feeling lucky. Counting n-grams across corpora is simple yet effective strategy to extract information from given texts.

We can generate n-grams by using an OrderedCollection lastWords to use as a buffer for last n words. We concatenate when buffer is full using join operator before adding it in n-grams. Define the following in the ChatFeatures class-side method.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

getNgramsFromLine: line withN: n
  "Returns an OrderedCollection of word ngrams with n = parameter n."

  | ngrams lastWords listOfWords |
  ngrams := OrderedCollection new.
  lastWords := OrderedCollection new.
  listOfWords := line splitOn: Character space.
  
  listOfWords do: [ :word |
    (lastWords size < n) ifFalse: [
      ngrams add: (' ' join: lastWords).
      lastWords removeFirst.
      ].
      lastWords addLast: word.
   ].
  ngrams add: (' ' join: lastWords).
  
  ^ ngrams

In line 12, we remove the first word from lastWords. Since n-grams act as a sliding window, we are emulating a queue in lastWords by adding at end and removing front. We also have added another ngram add at the end, to flush the buffer.

Here is how to use it:

1
2

line := 'A quick brown fox'.
ChatFeatures getNgramsFromLine: line withN: 2.

1

an OrderedCollection('A quick' 'quick brown' 'brown fox')

For character n-grams, we pass a String object instead of listOfWords, and join lastWords (or here, lastChars) using an empty string ''.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

getCharNgramsFromLine: line withN: n
  "Returns an OrderedCollection of character ngrams with n = parameter n."

  | ngrams lastChars |
  ngrams := OrderedCollection new.
  lastChars := OrderedCollection new.
  
  line do: [ :word |
    (lastChars size < n) ifFalse: [
      ngrams add: ('' join: lastChars).
      lastChars removeFirst.
      ].
      lastChars addLast: word.
   ].
  ngrams add: ('' join: lastChars).
  
  ^ ngrams

Here is how output will look for n=7 on I am lucky:

1

an OrderedCollection('I am lu' ' am luc' 'am luck' 'm lucky')

We now extend this to DataFrame, using the following code:

1
2
3
4
5
6
7

getNgramsFromDataFrame: df withN: n
  "Returns an array of word ngrams with n = parameter n."

  
  ^ ((df column: #Message) collect: [ :line |
     self getNgramsFromLine: line withN: n.
     ]) asArray.

You can create a similar one for char-ngrams.

Try running it as follows:

1

df addColumn: (ChatFeatures getNgramsFromDataFrame: df withN: 3) named: #Ngrams.

Output looks like this:

Punctuations makes inference difficult eg: index 19 has - which is counted as a word. For n-grams to be effective, we need to clean our DataFrame. Let’s add a few messages to make cleaning easier.

Setting appropriate column type

If you inspect the df, you’ll notice that all columns are of type ByteString, except a few cells with WideString. Columns like Date, Time need to be in appropriate type so that analyzing them becomes easier.

We iterate through rows, transforming each cell to appropriate data type:

1
2
3
4
5
6
7
8

setTypesFor: df
   "Transforms the df columns into appropriate types"

   df do: [ :row |
   row at: #Date transform: [ :date | date asDate ].
   row at: #Time transform: [ :time | time asTime ].
   row at: #Message transform: [ :message | message asWideString ].
   ].

This should be placed as a class-side method in ChatClenaer. Usage:

1

ChatCleaner setTypesFor: df.

An alternative way to implement this would be using toColumn: columnName applyElementwise: block method, or setting it in WhatsappReader itself.

Making messages lowercase

For analysing word count, we need to ensure that words are in similar case since 'Word' ~= 'word'. We apply a similar transformation lowercase:

1
2
3
4

messagesAsLowercase: df
   "Returns a DataSeries of the message column with lowercase strings"

   ^ ((df column: #Message) collect: [ :message | message asLowercase ]).

You can run it as:

1

df column: #Message put: (ChatCleaner messagesAsLowercase: df).

Keep words only

We need to remove punctuation so that when we split a string by space, we directly will get an array of words. The regex used to match non-words is [^\w\s]. ^ stands for not matching, \w stands for a series of letters, and \s stands for series of different spaces space \t \r \n.

1
2
3
4
5
6
7

getWordsFrom: df
   "Returns a DataSeries with removes punctuation and digits from #Message"

   ^ ((df column: #Message) collect: [ :message |
       message copyWithRegex: '[^\w\s]' matchesReplacedWith: ''
       ]
     )

Keep emojis only

To count emojis, we need a regex that will keep them only. However, their unicode ranges are too varied, and I found it easier just to match non-emoji characters and remove them using the regex [\w\d\s\\:.,''"-/?!()[]<>@’^“”=+_] (It’s just a regex with all symbols I have seen in chat mixed together).

1
2
3
4
5
6
7

getEmojisFrom: df
   "Returns the #Message column while removing non-emoji characters. The regex might not work perfectly"

   ^ ((df column: #Message) collect: [ :message |
       message copyWithRegex: '[\w\d\s\\:.,''"-/?!()[]<>@’^“”=+_]' matchesReplacedWith: ''
       ]
     )

Removing stopwords

Stopwords are the words that do not give you any insight, such as is, are, his, when etc. The list of stopwords here is taken from sklearn, who have sourced it from “Glasgow Information Retrieval Group”. Define following to class-side messages in ChatCleaner:

1
2
3
4
5
6
7

getStopwords
  "Returns array of stopwords"

  ^ Set withAll: #(
  'list' 'is' 'removed'
  'please' 'visit' github' 'repo' 'for' 'complete' 'list'
  )

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

removeStopwordsFrom: df
  "Removes stop words and returns Message dataseries"

  | stopWords words |
  stopWords := self getStopwords.
  ^ (df column: #Message) collect: [ :message |
   words := ((message trimBoth) splitOn: ('\s+' asRegex)).
   words := words reject: [ :word |
     stopWords includes: (word asLowercase)
      ].
   ' ' join: words
    ]

In the above snippet, we just reject input words present in stopWords and create an OrderedCollection out of those. To get words from message, we first remove any whitespace from both ends and the split using \s+ regex, which matches one or more whitespace characters.

Removing blacklisted messages

There are messages like This message was deleted and <Media omitted> which would not contribute to text analysis. To remove such messages (and any additional ones) we add them to a set and pass it to the following method:

1
2
3
4
5
6
7
8

removeMessages: blacklistSet from: df
   "Removes messages present in blacklistSet from df"

   | outputDf |
   outputDf := df reject: [ :row |
      blacklistSet includes: (row at: #Message)
      ].
   ^ outputDf   

This can be used as:

1
2
3
4

blacklist := Set withAll: #('This message was deleted' '<Media omitted>').
nGramDf := (ChatCleaner removeMessages: blacklist from: nGramDf).
" If you run this on group0.txt, you will see 17 messages removed. "
(df size) - (ngram size)

I think that’s all the messages needed! I’ll add few more if I think they are missing while writing next parts. You can find the messages implemented here: Whatsapp-Analyzer.

In the next part, we’ll analyze our parsed chat by using messages we have created.

If you are unfamiliar with Pharo, check out Pharo by Example as well as this blog post.

Prerequisites

1. Basics of Pharo/smalltalk

You must know the basics of Pharo, such as installing packages, defining classes/subclasses, message passing etc. These are well covered in the above linked blog post and Pharo by Example (first 3 chapters are enough for this post).

2. Whatsapp chat

The first thing we need is a Whatsapp Chat log. Open any Whatsapp group or personal chat, tap on options (3 dots on top right), then on more and select Export chat (without media). You may either save it to your Google Drive, email the chat to yourself or send it to someone in your contact and download it.

I have uploaded a sample Whatsapp group chat at this link, which you may use for following this tutorial. The sample has 247 lines from 50 users.

3. Pharo with Dataframe library

You need Pharo VM with Dataframe library installed. For this post, I used Pharo 7.0. You will find the instructions to install and use it in the book Pharo by Example.

Open playground, copy paste the following, and run it. It will install the Dataframe library. Save the image so that you won’t need to install it again.

1
2
3
4

Metacello new
  baseline: 'DataFrame';
  repository: 'github://PolyMathOrg/DataFrame/src';
  load.

An excellent resource for learning about this library is Dataframe by Example, which goes over the messages that you can use, and scenarios in which you might use them.

Reading the chat

Parsing lines

Here is how a line in the exported chat looks like:

1

4/10/19, 6:18 PM - Member14: Don't have a surname, it's empty in passport as well, what to fill in access via form in surname field, name?

We get four fields from a line - Date, Time, Author, and Message, in the following format:

1

<d/m/yy>, <h:mm (AM/PM)> - <author>: <message>

Date occurs till the first ,, time from , to -, author from - till second : (since time has a : in it) and message is the rest of the line.

However, a chat consists of non-message lines when a group name is changed, photo is updated, or members are removed or added, such as these:

1
2
3
4
5

4/10/19, 8:08 AM - Messages to this group are now secured with end-to-end encryption. Tap for more info.
3/28/19, 8:39 AM - Member0 created group "GROUPNAME"
4/10/19, 8:08 AM - Member0 added you
4/10/19, 8:29 AM - Member0 added Member4
4/10/19, 8:36 AM - Member0 added Member6

These can be detected as they (mostly) do not have more than one :. We ignore such lines since they are system generated and not users’ messages. Some of the messages may be multiline, such as:

1
2
3
4

4/10/19, 9:44 AM - Member5: Hi
Even I got the admit last night only.
Did go thru the links here, pretty informative.
So the next step is just to accept and send transcripts right? Anything else for the time being ?

An easy way to detect them is checking if a line starts with a date regex, \d(\d)?/\d(\d)?/\d\d. You can learn what regexes are online and test them at regex101. The lines that match regex should be added as a new row, and those which don’t should be appended to previous row.

We’ll use the above four fields as columns in our dataframe along with the checks discussed and parse the lines. It would be easier for us to parse if we define a custom class with appropriate messages. We create a class WhatsappReader inside WhatsappAnalyzer package.

1
2
3
4

DataFrameReader subclass: #WhatsappReader
  instanceVariableNames: ''
  classVariableNames: ''
  package: 'WhatsappAnalyzer'

The parent class is DataFrameReader since it allows us to use DataFrame readFrom: using: method. This is discussed in the next section. Add the following method into class-side:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

insert: line into: df
  "Parses given line and inserts it as a row in dataframe. An utility method."

  | commaIndex hyphenIndex colonIndex oldRow |
  (line isNotEmpty) ifTrue: [
    ((line copyUpTo: $,) matchesRegex: '\d(\d)?/\d(\d)?/\d\d') 
      ifTrue: [ 
        commaIndex := line indexOf: $,.
        hyphenIndex := line indexOf: $-.
        colonIndex := line indexOf: $: startingAt: hyphenIndex.
        (colonIndex ~= 0 ) ifTrue: [
          df addRow: { 
            line copyUpTo: $, .
            line copyFrom: commaIndex+2 to: hyphenIndex-2 .
            line copyFrom: hyphenIndex+2 to: colonIndex-1 .
            (line copyFrom: colonIndex+1 to: line size) allButFirst
          } named: df numberOfRows + 1.
        ]
      ]
      ifFalse: [
        oldRow := df row: df numberOfRows.
        oldRow at: 'Message' put: ((oldRow at: 'Message') , '\n' , line).
        df removeRow: df numberOfRows.
        df addRow: oldRow.
      ]
  ]

Line 6 checks if the message starts with a valid date, if it doesn’t, it appends the line to previous row using 21 to 24.

Line 11 serves as a check for valid message line - if the line doesn’t have a second :, it is a line that we are not interested in, such as when group name is changed.

You can try it out using the following code in Playground. Try all the possible lines (valid/invalid/multiline).

1
2
3
4

line := '4/10/19, 6:18 PM - Member14: Don''t have a surname, it''s empty in passport as well, what to fill in access via form in surname field, name?'.
df := DataFrame withColumnNames: #('Date' 'Time' 'Author' 'Message').
WhatsappReader insert: line into: df.
"Inspect df to see the change."

Reading the chat file

DataFrame library has a method readFrom: using: which enables us to write custom readers which can enter data into the dataframe. One such reader is DataFrameCsvReader whose implementation can be found here.

To achieve similar result, we have inherited the DataFrameReader class, which makes it readFrom: a subclass responsiblity. Now we need to override that method, and we do so as follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

readFrom: chatFileReference
  "Read Whatsapp chat and add it into the dataframe. Takes FileRefernce as it's argument"
  
  | df line |
  df := self createDataFrame.
  chatFileReference readStreamDo: [ :inputStream |
    [ inputStream atEnd ] whileFalse: [ 
      line := inputStream nextLine.
      self class insert: line into: df.
      ]
     ].
  ^ df

We read the FileReference line by line using readStreamDo and nextLine. inputStream atEnd ensures that we have read all lines. readStreamDo closes the stream when the block is executed. createDataFrame has been defined as follows:

1
2
3
4

createDataFrame 
  "Creates an empty dataframe with four columns parsed from Whatsapp chat"

  ^ DataFrame withColumnNames: #('Date' 'Time' 'Author' 'Message')

Now, we finally can run the program to get parsed chat.

1
2
3

fileRef := '/path/to/group0.txt' asFileReference.
reader := WhatsappReader new.
df := DataFrame readFrom: fileRef using: reader.

Inspecting df, we see the following: We were able to parse 220 lines of data (including multiline), and ignored 27 lines.

The resulting code is available at my GitHub: Whatsapp-Analyzer.

In the following parts, we’ll clean the data, find interesting metrics on it, and plot them!