pyreadstat.write_xport function behavior question

[wangw106]

I have a project and I use the pyreadstat.read_xport() to read SAS xpt files (version 5) to pandas dataframe, process the dataframe, e.g. standardize some of the column values, then write the processed dataframe to the SAS xpt files by using the pyreadstat.write_xport() in a different folder.

The pyreadstat.read_xport() works perfectly. The pyreadstat.write_xport() also works great except one behavior I don't understand:
Given an example, I am processing a ts.xpt file, the program reads the source file with a size of 10k, shrinks one of column values, e.g, the column previous has 8 characters, after processing, it only has 6 characters. then write the processed dataframe to ts.xpt in a different folder. Now the new ts.xpt file has a size of < 10k. At this point, everything works perfectly.

However, if my target folder already has a ts.xpt file (call this old ts.xpt) with a size larger than the new ts.xpt file, the new ts.xpt file created by pyreadstat.write_xport() will have the same size as old ts.xpt, with all the content from the new ts.xpt, followed by some "garbage" from the old ts.xpt.

Can anybody explain why does the pyreadstat.write_xport() behave like this? I am hope the function will overwrite even if a file with same filename exists.

Thanks in advance.

[ofajardo]

Please provide a way to replicate your issue, I don't see such behavior. Also, in which operating system do you work?

[wangw106]

Here is how to duplicate this issue:

Pretty much you can use any dataset, here it ts.xpt file I used in the below script (https://www.pdslifesciences.com/send-dataset).
Extract the ts.xpt file and put in you TEMP folder. The ts.xpt file has 30 records.
I am running in windows 10, with python 3.10. use the pyreadstat of 1.1.6

xpt_df, meta = pyreadstat.read_xport("C:/TEMP/ts.xpt", metadataonly=False, encoding="ascii")
#remove 3 rows from the dataframe
dfTS = xpt_df[-xpt_df["TSPARMCD"].isin(["ROUTE", "SPECIES", "STRAIN"])]

#below code works, as the file ts1.xpt does not exist in folder, the ts1.xpt file has 27 records
pyreadstat.write_xport(dfTS, "C:/TEMP/ts1.xpt",
file_format_version=5,
table_name="TS",
file_label="Trial Summary",
column_labels=meta.column_labels
)

#below code created ts.xpt file will have garbage at the end of the file, as the ts.xpt exists in the folder. The first 27 records are correct, the last 3 records are garbage.
pyreadstat.write_xport(dfTS, "C:/TEMP/ts.xpt",
file_format_version=5,
table_name="TS",
file_label="Trial Summary",
column_labels=meta.column_labels
)

[ofajardo]

it's a windows specific issue. I'll transfer it to an issue.