Sorting function in Pandas, returns messy data

Question

I am trying to sort data in a CSV file using sort function in Pandas using the following code. I have 229 rows in original file. But the output of sorting is 245 rows, because some of the data in a field were printed in the next row and some of the rows do not have any value.

sample=pd.read_csv("sample.csv" , encoding='latin-1', skipinitialspace=True)
sample_sorted = sample.sort_values(by = ['rating'])
sample_sorted.to_csv("sample_sorted.csv")

I think, this problem happened because in some cells data was entered by generating new lines. For example this is the content of a cell in original file. When I sort the original file, the second line was printed in a new row and 3 rows left empty between first and second line.

"Side effects are way to extreme. 



E-mail me if you have experianced the same things."

Any suggestion? Thanks !

@Merlin, I thought it maybe some other character inside the file, such as arabic characters. yes the file has header. — Mary
– Mary, Commented Sep 5, 2016 at 21:26
@Mary, that's interesting. Can you upload your CSV file somewhere, so we can reproduce your issue? — MaxU - stand with Ukraine
– MaxU - stand with Ukraine, Commented Sep 5, 2016 at 21:27
@Mary I understand the nature of the content is sensitive. However, if this is an important problem to solve, it may be worth while to create a fake file that produces the same issue you are observing. This way, you can share it with us. The issue can probably be recreated with only a few rows. This would help you get your answer quicker. — piRSquared
– piRSquared, Commented Sep 5, 2016 at 22:15

binaryaaron · Accepted Answer · 2016-09-05 23:12:49Z

2

You could try to remove the newlines in your problem column.

sample=pd.read_csv("sample.csv" , encoding='latin-1', skipinitialspace=True)
sample["problem_column"] = (sample["problem_column"].
                            apply(lambda x: " ".join([word for word in x.split()])
                            )

and see if that helps at all. It's difficult to see why that's happening without a reproducible sample.

answered Sep 5, 2016 at 23:12

binaryaaron

6517 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Sorting function in Pandas, returns messy data

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related