CASE STUDY 2 - THE SOCIAL SURVEY

By Nino Miljkovic

---

This case study expands on the Social Survey analysis, originally introduced in the Chapter 14 of Murach’s 'Python for Data Science'. The Exercise 14-1 on page 508 guides us through 3 steps of the analysis:

1. Get the data

2. Clean the data and prepare the data

3. Form and test a hypothesis

---

1. Get the Data

Necessary modules:

import pandas as pd
import seaborn as sns
import pyreadstat

Open the Codebook pdf and read through the question sets that start on pages 2,715 and 2,768. Run the cells that get the data for the DataFrames named socialMedia and mentalHealth:

socialMedia = pd.read_stata('GSS7218_R3.DTA', 
                            columns=['id','snsmyear','intwkdyh','intwkdym','intwkenh','intwkenm'])

socialMedia.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 64814 entries, 0 to 64813
Data columns (total 6 columns):
 #   Column    Non-Null Count  Dtype   
---  ------    --------------  -----   
 0   id        64814 non-null  int16   
 1   snsmyear  1120 non-null   category
 2   intwkdyh  1220 non-null   category
 3   intwkdym  1224 non-null   category
 4   intwkenh  1216 non-null   category
 5   intwkenm  1225 non-null   category
dtypes: category(5), int16(1)
memory usage: 447.1 KB

mentalHealth = pd.read_stata('GSS7218_R3.DTA', 
                             columns=['id','cesd1','cesd2','cesd3','cesd4','cesd5'])

mentalHealth.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 64814 entries, 0 to 64813
Data columns (total 6 columns):
 #   Column  Non-Null Count  Dtype   
---  ------  --------------  -----   
 0   id      64814 non-null  int16   
 1   cesd1   963 non-null    category
 2   cesd2   965 non-null    category
 3   cesd3   966 non-null    category
 4   cesd4   964 non-null    category
 5   cesd5   964 non-null    category
dtypes: category(5), int16(1)
memory usage: 444.2 KB

For both DataFrames, drop all rows that have missing values, and set an index on the id column:

socialMedia = socialMedia.dropna().set_index('id')

mentalHealth = mentalHealth.dropna().set_index('id')

socialMedia.head() # verifying 'id' is index

	snsmyear	intwkdyh	intwkdym	intwkenh	intwkenm
id
5	2016.0	1.0	0.0	1.0	0.0
6	2014.0	1.0	0.0	0.0	30.0
8	2010.0	2.0	0.0	3.0	0.0
10	2011.0	6.0	0.0	5.0	0.0
11	2015.0	7.0	0.0	4.0	0.0

socialMedia.info()  # verifying missing-value rows were dropped

<class 'pandas.core.frame.DataFrame'>
Index: 1112 entries, 5 to 2867
Data columns (total 5 columns):
 #   Column    Non-Null Count  Dtype   
---  ------    --------------  -----   
 0   snsmyear  1112 non-null   category
 1   intwkdyh  1112 non-null   category
 2   intwkdym  1112 non-null   category
 3   intwkenh  1112 non-null   category
 4   intwkenm  1112 non-null   category
dtypes: category(5)
memory usage: 11.5 KB

mentalHealth.head() # verifying 'id' is index

	cesd1	cesd2	cesd3	cesd4	cesd5
id
3	none or almost none of the time	most of the time	most of the time	none or almost none of the time	none or almost none of the time
5	none or almost none of the time	some of the time	most of the time	none or almost none of the time	some of the time
8	none or almost none of the time	some of the time	all or almost all of the time	none or almost none of the time	some of the time
10	none or almost none of the time	some of the time	most of the time	none or almost none of the time	none or almost none of the time
14	some of the time	none or almost none of the time	some of the time	none or almost none of the time	some of the time

mentalHealth.info()  # verifying missing-value rows were dropped

<class 'pandas.core.frame.DataFrame'>
Index: 961 entries, 3 to 2867
Data columns (total 5 columns):
 #   Column  Non-Null Count  Dtype   
---  ------  --------------  -----   
 0   cesd1   961 non-null    category
 1   cesd2   961 non-null    category
 2   cesd3   961 non-null    category
 3   cesd4   961 non-null    category
 4   cesd5   961 non-null    category
dtypes: category(5)
memory usage: 7.6 KB

Use an inner join to join the two DataFrames, and assign the new DataFrame to a variable named socialHealth:

socialHealth = socialMedia.join(mentalHealth, how='inner')

Display the first five rows and use the info() method to view the data types for each column:

socialHealth.head()

	snsmyear	intwkdyh	intwkdym	intwkenh	intwkenm	cesd1	cesd2	cesd3	cesd4	cesd5
id
5	2016.0	1.0	0.0	1.0	0.0	none or almost none of the time	some of the time	most of the time	none or almost none of the time	some of the time
8	2010.0	2.0	0.0	3.0	0.0	none or almost none of the time	some of the time	all or almost all of the time	none or almost none of the time	some of the time
10	2011.0	6.0	0.0	5.0	0.0	none or almost none of the time	some of the time	most of the time	none or almost none of the time	none or almost none of the time
22	2009.0	1.0	0.0	2.0	0.0	none or almost none of the time	none or almost none of the time	most of the time	none or almost none of the time	none or almost none of the time
25	2007.0	10.0	0.0	8.0	0.0	none or almost none of the time	some of the time	most of the time	some of the time	some of the time

socialHealth.info()

<class 'pandas.core.frame.DataFrame'>
Index: 544 entries, 5 to 2867
Data columns (total 10 columns):
 #   Column    Non-Null Count  Dtype   
---  ------    --------------  -----   
 0   snsmyear  544 non-null    category
 1   intwkdyh  544 non-null    category
 2   intwkdym  544 non-null    category
 3   intwkenh  544 non-null    category
 4   intwkenm  544 non-null    category
 5   cesd1     544 non-null    category
 6   cesd2     544 non-null    category
 7   cesd3     544 non-null    category
 8   cesd4     544 non-null    category
 9   cesd5     544 non-null    category
dtypes: category(10)
memory usage: 11.3 KB

2. Clean and Prepare the Data

Run the cell to rename the columns to make them more readable:

socialHealth.rename(columns={'snsmyear':'yearAquired', 
                             'intwkdyh':'weekdayHours', 
                             'intwkdym':'weekdayMinutes', 
                             'intwkenh':'weekendHours', 
                             'intwkenm':'weekendMinutes', 
                             'cesd1':'depressed', 
                             'cesd2':'restlessSleep', 
                             'cesd3':'happy', 
                             'cesd4':'lonely', 
                             'cesd5':'sad'}, 
                    inplace=True)

View the value counts for the weekdayMinutes and weekendMinutes columns:

socialHealth.weekdayMinutes.value_counts()

weekdayMinutes
0.0     451
30.0     67
15.0      8
20.0      8
45.0      5
10.0      3
2.0       1
59.0      1
1.0       0
3.0       0
5.0       0
25.0      0
40.0      0
55.0      0
Name: count, dtype: int64

socialHealth.weekendMinutes.value_counts()

weekendMinutes
0.0     448
30.0     61
10.0      7
15.0      7
45.0      7
20.0      6
3.0       2
5.0       2
40.0      2
1.0       1
59.0      1
25.0      0
35.0      0
50.0      0
Name: count, dtype: int64

Remove any rows that don’t have values of 0, 15, 30, or 45 in these columns:

socialHealth = socialHealth.query(
    'weekdayMinutes in [0,15,30,45] and weekendMinutes in [0,15,30,45]')

Convert the first five columns from the category type to the int type:

cols = ['yearAquired', 'weekdayHours', 'weekdayMinutes', 'weekendHours', 'weekendMinutes']
socialHealth[cols] = socialHealth[cols].astype(int)

socialHealth.info()

<class 'pandas.core.frame.DataFrame'>
Index: 517 entries, 5 to 2867
Data columns (total 10 columns):
 #   Column          Non-Null Count  Dtype   
---  ------          --------------  -----   
 0   yearAquired     517 non-null    int64   
 1   weekdayHours    517 non-null    int64   
 2   weekdayMinutes  517 non-null    int64   
 3   weekendHours    517 non-null    int64   
 4   weekendMinutes  517 non-null    int64   
 5   depressed       517 non-null    category
 6   restlessSleep   517 non-null    category
 7   happy           517 non-null    category
 8   lonely          517 non-null    category
 9   sad             517 non-null    category
dtypes: category(5), int64(5)
memory usage: 24.7 KB

Modify the weekdayHours and weekendHours columns so they store the combined hours and minutes. To do that, you’ll need to divide the minutes data by 60 and add it to the hours. Drop the weekdayMinutes and weekendMinutes columns since you don’t need them anymore:

socialHealth.weekdayHours = socialHealth.weekdayHours + (socialHealth.weekdayMinutes / 60)
socialHealth.weekendHours = socialHealth.weekendHours + (socialHealth.weekendMinutes / 60)

socialHealth = socialHealth.drop(columns=['weekdayMinutes','weekendMinutes'])

socialHealth.head()  # displaying the first five rows

	yearAquired	weekdayHours	weekendHours	depressed	restlessSleep	happy	lonely	sad
id
5	2016	1.0	1.0	none or almost none of the time	some of the time	most of the time	none or almost none of the time	some of the time
8	2010	2.0	3.0	none or almost none of the time	some of the time	all or almost all of the time	none or almost none of the time	some of the time
10	2011	6.0	5.0	none or almost none of the time	some of the time	most of the time	none or almost none of the time	none or almost none of the time
22	2009	1.0	2.0	none or almost none of the time	none or almost none of the time	most of the time	none or almost none of the time	none or almost none of the time
25	2007	10.0	8.0	none or almost none of the time	some of the time	most of the time	some of the time	some of the time

3. Form and Test a Hypothesis

---

Hypothesis: More weekday social media use is correlated with lower sleep quality.

---

Use Seaborn to create a box plot for the restlessSleep and weekdayHours columns:

graph = sns.catplot(data=socialHealth, x='restlessSleep', y='weekdayHours', kind='box')

for ax in graph.axes.flat:
    ax.tick_params('x',labelrotation = 90)

Use the cut() method to bin the weekdayHours column into five 3-hour groups (0-3 hours, 3-6 hours, etc.). Use the labels parameter to create intuitive labels for your groups. Assign the bins to a new column called weekdayHoursBin:

socialHealth['weekdayHoursBin'] = pd.cut(socialHealth.weekdayHours, bins=[0,3,6,9,12,15], 
                                           labels=['0-3','3-6','6-9','9-12','12-15'], right=False)

Display the value counts for all combinations of the weekdayHoursBin and restlessSleep columns:

socialHealth[['weekdayHoursBin','restlessSleep']].value_counts(sort=False)

weekdayHoursBin  restlessSleep                  
0-3              none or almost none of the time     94
                 some of the time                   122
                 most of the time                    49
                 all or almost all of the time       24
3-6              none or almost none of the time     28
                 some of the time                    61
                 most of the time                    27
                 all or almost all of the time       11
6-9              none or almost none of the time     22
                 some of the time                    20
                 most of the time                     8
                 all or almost all of the time        4
9-12             none or almost none of the time      9
                 some of the time                    11
                 most of the time                     2
                 all or almost all of the time        1
12-15            none or almost none of the time      2
                 some of the time                     5
                 most of the time                     8
                 all or almost all of the time        4
Name: count, dtype: int64

Use Seaborn to create a count plot that shows the count of the restlessSleep values for each time bin:

sns.catplot(data=socialHealth, kind='count', x='weekdayHoursBin', hue='restlessSleep')

<seaborn.axisgrid.FacetGrid at 0x16214b5d0>

Group the DataFrame by the weekdayHoursBin column. Then, call the value_ counts() method from the restlessSleep column with the normalize parameter set to True. Convert the results to a DataFrame and assign it to a new variable named hoursGrouped:

hoursGrouped = socialHealth.groupby('weekdayHoursBin', observed=True).restlessSleep.value_counts(
    normalize=True).to_frame()

Rename the column for the hoursGrouped DataFrame to percentage and reset the index:

hoursGrouped.head()  # checking the name of the column for hoursGrouped

		proportion
weekdayHoursBin	restlessSleep
0-3	some of the time	0.422145
	none or almost none of the time	0.325260
	most of the time	0.169550
	all or almost all of the time	0.083045
3-6	some of the time	0.480315

hoursGrouped.rename(columns={'proportion': 'percentage'}, inplace=True) # renaming the column
hoursGrouped.reset_index(inplace=True)                                  # resetting the index

hoursGrouped.head()

	weekdayHoursBin	restlessSleep	percentage
0	0-3	some of the time	0.422145
1	0-3	none or almost none of the time	0.325260
2	0-3	most of the time	0.169550
3	0-3	all or almost all of the time	0.083045
4	3-6	some of the time	0.480315

Use Seaborn to create a bar plot that displays the percentage on the y-axis and the weekdayHoursBin column on the x-axis. In addition, this plot should use colors to identify the restlessSleep column:

sns.catplot(data=hoursGrouped, kind='bar', x='weekdayHoursBin', y='percentage', hue='restlessSleep')

<seaborn.axisgrid.FacetGrid at 0x161d30790>

Conclusion Statement:

After normalizing the restless sleep parameters we got a clearer picture of how social media usage affects sleep patterns. As the last graph shows, the correlation is apparent, however, only in the case of social media overuse (12-15 hours). In other cases there is no clear indication that it impacts the quality of sleep since the blue bar ('none or almost none of the time') is strongly represented, while the red ('all or almost all of the time') has the lowest percentages.

---

Develop your own hypothesis:

---

MY HYPOTHESIS: Social media use can lead to depression

---

socialHealth.head() 

	yearAquired	weekdayHours	weekendHours	depressed	restlessSleep	happy	lonely	sad	weekdayHoursBin
id
5	2016	1.0	1.0	none or almost none of the time	some of the time	most of the time	none or almost none of the time	some of the time	0-3
8	2010	2.0	3.0	none or almost none of the time	some of the time	all or almost all of the time	none or almost none of the time	some of the time	0-3
10	2011	6.0	5.0	none or almost none of the time	some of the time	most of the time	none or almost none of the time	none or almost none of the time	6-9
22	2009	1.0	2.0	none or almost none of the time	none or almost none of the time	most of the time	none or almost none of the time	none or almost none of the time	0-3
25	2007	10.0	8.0	none or almost none of the time	some of the time	most of the time	some of the time	some of the time	9-12

socialHealth.info() 

<class 'pandas.core.frame.DataFrame'>
Index: 517 entries, 5 to 2867
Data columns (total 9 columns):
 #   Column           Non-Null Count  Dtype   
---  ------           --------------  -----   
 0   yearAquired      517 non-null    int64   
 1   weekdayHours     517 non-null    float64 
 2   weekendHours     517 non-null    float64 
 3   depressed        517 non-null    category
 4   restlessSleep    517 non-null    category
 5   happy            517 non-null    category
 6   lonely           517 non-null    category
 7   sad              517 non-null    category
 8   weekdayHoursBin  512 non-null    category
dtypes: category(6), float64(2), int64(1)
memory usage: 17.4 KB

socialHealth.depressed.value_counts()

depressed
none or almost none of the time    345
some of the time                   146
most of the time                    21
all or almost all of the time        5
Name: count, dtype: int64

Converting 'depressed' categorical responses to numerical values via map function:

depression_mapping = {
    "none or almost none of the time": 0,
    "some of the time": 1,
    "most of the time": 2,
    "all or almost all of the time": 3
}
socialHealth['depression_level'] = socialHealth['depressed'].map(depression_mapping)

socialHealth.head()

	yearAquired	weekdayHours	weekendHours	depressed	restlessSleep	happy	lonely	sad	weekdayHoursBin	depression_level
id
5	2016	1.0	1.0	none or almost none of the time	some of the time	most of the time	none or almost none of the time	some of the time	0-3	0
8	2010	2.0	3.0	none or almost none of the time	some of the time	all or almost all of the time	none or almost none of the time	some of the time	0-3	0
10	2011	6.0	5.0	none or almost none of the time	some of the time	most of the time	none or almost none of the time	none or almost none of the time	6-9	0
22	2009	1.0	2.0	none or almost none of the time	none or almost none of the time	most of the time	none or almost none of the time	none or almost none of the time	0-3	0
25	2007	10.0	8.0	none or almost none of the time	some of the time	most of the time	some of the time	some of the time	9-12	0

Creating plots that show the relationship between depression levels and the average social media usage:

# Bar plot showing average social media usage per depression level
sns.barplot(x='depression_level', y='weekdayHours', data=socialHealth)

<Axes: xlabel='depression_level', ylabel='weekdayHours'>

# Box plot showing Social media use (weekdayHours) vs Depression levels
sns.boxplot(x='depression_level', y='weekdayHours', data=socialHealth)

<Axes: xlabel='depression_level', ylabel='weekdayHours'>

Converting ' happy' categorical responses to numerical values to contrast it to 'depression' results:

socialHealth.happy.value_counts()

happy
most of the time                   277
all or almost all of the time      126
some of the time                   102
none or almost none of the time     12
Name: count, dtype: int64

happiness_mapping = {
    "none or almost none of the time": 0,
    "some of the time": 1,
    "most of the time": 2,
    "all or almost all of the time": 3
}
socialHealth['happiness_level'] = socialHealth['happy'].map(happiness_mapping)

Creating plots to show the relationship between the levels of happines and social media usage:

# Box plot showing Social media use (weekdayHours) vs Happiness levels
sns.boxplot(x='happiness_level', y='weekdayHours', data=socialHealth)

<Axes: xlabel='happiness_level', ylabel='weekdayHours'>

# Bar plot showing average social media usage per happiness level
sns.barplot(x='happiness_level', y='weekdayHours', data=socialHealth)

<Axes: xlabel='happiness_level', ylabel='weekdayHours'>

Conclusion Statement:

The analysis suggests a potential link between high social media usage and increased depression levels, as individuals with higher depression levels tend to have a greater social media usage and more extreme outliers. In contrast, happiness levels do not show a strong correlation with social media use. Their average usage remains fairly stable across different happiness groups.

The findings are unclear on whether the social media use is directly influencing the rise of depression, or rather serving as a coping mechanism for those that are experiencing it. However, while the causality is not proven, the correlation is visible.

On the other hand, happiness does not seem to be significantly impacted by social media use, reinforcing the idea that depression and happiness are influenced by different factors.