Python Pandas: Efficiently Assign Values To A Slice
I have a dataframe
next_train with weekly data for many players (80,000 players observed through 4 weeks, total of 320,000 observations) and a dictionary
players containing a binary variable for some of the players (say 10,000). I want to add this binary variable to the dataframe
next_train (if a player is not in the dictionary
players, I set the variable equal to zero). This is how I'm doing it:
next_train = pd.read_csv() # ... calculate dictionary 'players' ... next_train['variable'] = 0 for player in players: next_train.loc[next_train['id_of_player'] == player, 'variable'] = players[player]
for loop takes ages to complete, and I don't understand why. It looks like the task is to perform binary search for the value
player in my dataframe for 10,000 times (size of the
players dictionary), but the execution time is several minutes. Is there any efficient way to do this task?
You should use
map instead of slicing, that will be way faster:
next_train['variable'] = next_train.id_of_player.map(players)
As you want 0 in the other rows, you can then use
next_train.variable.fillna(0,inplace = True)
Moreover, if your dictionnary only contains boolean values, you might want to redefine the type of
variable column to take less space. So you end with this piece of code:
next_train['variable'] = next_train.id_of_player.map(players).fillna(0).astype(int)
- → What are the pluses/minuses of different ways to configure GPIOs on the Beaglebone Black?
- → Django, code inside <script> tag doesn't work in a template
- → React - Django webpack config with dynamic 'output'
- → GAE Python app - Does URL matter for SEO?
- → Put a Rendered Django Template in Json along with some other items
- → session disappears when request is sent from fetch
- → Python Shopify API output formatted datetime string in django template
- → Shopify app: adding a new shipping address via webhook
- → Shopify + Python library: how to create new shipping address
- → shopify python api: how do add new assets to published theme?
- → Access 'HTTP_X_SHOPIFY_SHOP_API_CALL_LIMIT' with Python Shopify Module