0

I want to process each row of a csv file and store into database. After that row is stored I want to delete that row in the csv file. I have written code for fetching and storing the row. But I got stuck on deleting a row on csv file.

My code:

import csv
  entries = csv.reader(open('/opt/workspace/jup/juppro/media/docs/phonebook.csv', 'rb'), dialect='excel-tab')
  entries_list = []
  entries_list.extend(entries)
  total_records= len(entries_list)
  uploaded_records=0
  for data in entries_list:
      uploaded_records=uploaded_records+1
      cols = data[0].replace('"','')
      cols1= cols.split(';')
      contact = phonebook()
      contact.email = cols1[0]
      contact.firstname = cols1[1]
      contact.lastname = cols1[2]
      contact.phoneno = cols1[3]
      #names.append(data[0])
      contact.save()
Improve this question
4
  • If you process each row and then delete each row, there will be no more csv file at the end. So why do you want to delete each row one after the other instead of deleting the entire file when it will have been entirely treated ? Commented Mar 9, 2011 at 9:56
  • @eyquem: actually i want to know each time the progress of storing the entire csv rows in to database. For that what i planned to do is calling a function which store 1 row at time to database,delete that row from csv and then increase the count. Then i can find the progress by rowsstored*100/totalrows. Commented Mar 9, 2011 at 9:59
  • Your aim is only to be informed of the progression of treatment ? I suggest you to see the function enumerate() in the docs. Will the file be treated always entirely ? Commented Mar 9, 2011 at 10:04
  • "deleting a row or line in a file" doesn't mean any concrete thing. Deleting can't be done by removing chunks of bits in a hard disk. The effect of deleting must be obtained by controled rewriting Commented Mar 9, 2011 at 10:06

2 Answers 2

Reset to default
1
  1. Read the CSV into memory e.g. a list.
  2. Save the relevant rows to the database and remove them from the list.
  3. Save the remaining rows in the list back to the CSV file.
Improve this answer
Sign up to request clarification or add additional context in comments.

7 Comments

@mdm thanks for answer...but that csv file has more than 500000 records. is it right to store it in list
In that case, maybe not. You could write to a new CSV file as you are reading from the original (writing those rows that you don't want to delete), which would eliminate the need to store it in memory.
@mdm actually my problem is... a ajax function will call python method. that python method will store the first row of csv and delete that row... I will call this ajax funtion repeatedly...
Is this some kind of import then? Why can't you just run the routine for all rows in the CSV all in one go?
@mdm. this is not import.the objective is to upload the csv to database with the progress shown.
|
0

I propose you this code

import csv
from os.path import getsize
from itertools import islice

print 'size of file :',getsize('/opt/workspace/jup/juppro/media/docs/phonebook.csv')

entries = csv.reader(open('/opt/workspace/jup/juppro/media/docs/phonebook.csv', 'rb'), dialect='excel-tab')

n = 3
cnt = 0
chunk = True
while chunk:
    chunk = list(islice(entries,n))
    if chunk==[]: 
        print 'processing ended'
        break
    # here: all the treatments of the rows being in chunk
    cnt += n
    print cnt

If all the file is treated, you don't care of deleting the rows while processing: you will delete the file after the treatment.

If you want to delete some rows on the basis of precise conditions, say it and we will adapt the above code

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.