Ad
How To Count The Number Of Repetation Of Words And Assign A Number And Append Into Dataframe
I am having a dataset of all the abstracts and the author gender. Now i want to get the all the repetitions of words gender wise so that i can plot it as a graph number of repetition of words with respect to gender.
data_path = '/content/digitalhumanities - forum-and-fiction.csv'
def change_table(data_path):
df = pd.read_csv(data_path)
final = df.drop(["Title", "Author", "Season", "Year", "Keywords", "Issue No", "Volume"], axis=1)
fin = final.set_index('Gender')
return fin
change_table(data_path).T
This is the out put i got
| Gender | None | Female | Male | None | None | Male ,Female |None | Male ,Female |
|:----------|---------------------------------------------------|---------------------------------------------------|---------------------------------------------------|------------|---------------------------------------|---------------------------------------------------|---------------------------------------------------|---------------------------------------------------|---------------------------------------------------|---------------------------------------------------:|
| Abstract | This article describes Virginia Woolf's preocc... | The Amazonian region occupies a singular place... | This article examines Kipling's 1901 novel Kim... | Pamela; or | Virtue Rewarded uses a literary fo... | This article examines Nuruddin Farah's 1979 no... | Ecological catastrophe has challenged the cont... | British political fiction was a satirical genr... | The Lydgates have bought too much furniture an...
Now how can i get the repetition of each word in the abstract with respect to gender and append to the data frame.
Expecting output example
|gender|male|female|none|
|------|----|------|----|
| This | 3| 0| 0|
| occupies | 5| 3| 0|
| examines | 6| 0| 0|
| British | 0| 0| 7|
Ad
Answer
Use crosstab
with splitting stacked values by DataFrame.stack
:
#removed T
df = change_table(data_path)
#reshape with split columns
df = (df.stack()
.rename_axis(('Type','Gender'))
.str.split(expand=True)
.stack()
.reset_index(name='Word'))
#explode Type by split with ,
df = df.assign(Type = df['Type'].str.split(',')).explode('Type')
#remove stpowords
from nltk.corpus import stopwords
stop_words = set(stopwords.words('english'))
df = df[~df['Word'].isin(stop_words)]
#get counts per Gender, Word and Type
df1 = pd.crosstab([df['Gender'], df['Word']], df['Type']).reset_index()
#or get counts per Word and Type
df2 = pd.crosstab([df['Word'], df['Type'])
Ad
source: stackoverflow.com
Related Questions
- → What are the pluses/minuses of different ways to configure GPIOs on the Beaglebone Black?
- → Django, code inside <script> tag doesn't work in a template
- → React - Django webpack config with dynamic 'output'
- → GAE Python app - Does URL matter for SEO?
- → Put a Rendered Django Template in Json along with some other items
- → session disappears when request is sent from fetch
- → Python Shopify API output formatted datetime string in django template
- → Can't turn off Javascript using Selenium
- → WebDriver click() vs JavaScript click()
- → Shopify app: adding a new shipping address via webhook
- → Shopify + Python library: how to create new shipping address
- → shopify python api: how do add new assets to published theme?
- → Access 'HTTP_X_SHOPIFY_SHOP_API_CALL_LIMIT' with Python Shopify Module
Ad