Resolving Issue With "Copy Of A Slice From A DataFrame"

- 1 answer

below is a code snipped, I like to explain alongside with comments. The main purpose of the script: Open/create a csv and continue a script at the last position. While the script is running, make sure to write row by row into the Dataframe, so that none of the processed data is being lost.

data = pd.read_csv('input.csv', sep=',')


with open('output.csv', 'a+') as f:

# Continue script where it stopped last
pos = len(f.readlines()[1:])
data = data.iloc[pos:]

    # Get two values from a function, write into two columns
    # progress_apply shows the progress (can be improved though)
    data[['city', 'country']] = data.progress_apply(func, axis=1, result_type='expand')
    # Append data to output.csv, but only add header for first entry
    data.to_csv(f, mode='a', header=f.tell()==0, encoding='utf-8', index=False)

    print('Error at position {}.'.format(pos))

When the script finished processing, it prints:

SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame. Try using .loc[row_indexer,col_indexer] = value instead

How can I resolve the issue, i.e. what's wrong with the code? Thanks for your help!



It is a warning due to the positional argument and is causing a chained assignment, typically, the warning indicates that operation might not have worked as expected. You can set the warning off by adding the following line right before the loc[:],

pd.set_option('mode.chained_assignment', None)