1

Firstly, I have searched for related answers, there is a few - but couldn't find anything quite specific to what I am looking for. I am sure there is a better method!

Basically, I am looking for some advice, I have a script in which is converting a CSV file which is on the server into a MYSQL database. However, the CSV file is 20mb and the script is just timing out. I have modified my php.ini so it shouldn't time out, but instead it causes a 500 internal server error. Is there a way I can keep the page live and not time out (would jquery do something like this) - If not, I know I have used like shell execution for large SQL files in the past, is there an alternative to use something like this? advice would really be appreciated. My programming knowledge isnt really too good, so please be patient

Here is the script I am using:

<?php 
$databasehost = "localhost";
$databasename = "test";
$databasetable = "sample";
$databaseusername ="test";
$databasepassword = "";
$fieldseparator = ",";
$lineseparator = "\n";
$csvfile = "filename.csv";
/********************************/
/* Would you like to add an ampty field at the beginning of these records?
/* This is useful if you have a table with the first field being an auto_increment integer
/* and the csv file does not have such as empty field before the records.
/* Set 1 for yes and 0 for no. ATTENTION: don't set to 1 if you are not sure.
/* This can dump data in the wrong fields if this extra field does not exist in the table
/********************************/
$addauto = 0;
/********************************/
/* Would you like to save the mysql queries in a file? If yes set $save to 1.
/* Permission on the file should be set to 777. Either upload a sample file through ftp and
/* change the permissions, or execute at the prompt: touch output.sql && chmod 777 output.sql
/********************************/
$save = 1;
$outputfile = "output.sql";
/********************************/


if(!file_exists($csvfile)) {
      echo "File not found. Make sure you specified the correct path.\n";
      exit;
}

$file = fopen($csvfile,"r");

if(!$file) {
      echo "Error opening data file.\n";
      exit;
}

$size = filesize($csvfile);

if(!$size) {
      echo "File is empty.\n";
      exit;
}

$csvcontent = fread($file,$size);

fclose($file);

$con = @mysql_connect($databasehost,$databaseusername,$databasepassword) or die(mysql_error());
@mysql_select_db($databasename) or die(mysql_error());

$lines = 0;
$queries = "";
$linearray = array();

foreach(split($lineseparator,$csvcontent) as $line) {

      $lines++;

      $line = trim($line," \t");

      $line = str_replace("\r","",$line);

      /************************************
      This line escapes the special character. remove it if entries are already escaped in the csv file
      ************************************/
      $line = str_replace("'","\'",$line);
      /*************************************/

      $linearray = explode($fieldseparator,$line);

      $linemysql = implode("','",$linearray);

      if($addauto)
            $query = "insert into $databasetable values('','$linemysql');";
      else
            $query = "insert into $databasetable values('$linemysql');";

      $queries .= $query . "\n";

      @mysql_query($query);
}

@mysql_close($con);

if($save) {

      if(!is_writable($outputfile)) {
            echo "File is not writable, check permissions.\n";
      }

      else {
            $file2 = fopen($outputfile,"w");

            if(!$file2) {
                  echo "Error writing to the output file.\n";
            }
            else {
                  fwrite($file2,$queries);
                  fclose($file2);
            }
      }

}

echo "Found a total of $lines records in this csv file.\n";


?>
1
  • Learn about using functions like fgetcsv() Commented Mar 17, 2012 at 19:46

2 Answers 2

2

Rather than process the csv manually, you should be able to do it in one MySQL command using LOAD DATA INFILE. This will shift the processing to MySQL and should mean PHP doesn't time out. If it is still timing out you can always run the same command from the shell.

Sign up to request clarification or add additional context in comments.

3 Comments

Pardon my lack of knowledge, but can a LOAD DATA INFILE be written in PHP?
@BushellConsultancy Sure, it's just a MySQL command, you should be able to run it on your database server using any of the PHP-MySQL libraries.
Much appreciated, with a bit of searching based on your result I got the job done. In case anyone else wants to know: $query = "LOAD DATA LOCAL INFILE 'aw2_dap_combined_all_csv.csv' INTO TABLE mobile FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n'";
1

Don't parse your own CSV

Use PHP's built-in function fgetcsv()

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.