0

I have a website where users can upload a csv file. Each file might have a different number of columns and rows and I use the following to handle the upload of a csv file and then write this data into a new table. This all works but it can be slow if the csv file is large. Writing the data row by row is slow and I wonder if there is a way of writing the whole csv file ($matrix - see below) into the newly created SQL table in one go?

//Handling the csv file and reading data into $matrix
$file = $_FILES['csv1']['tmp_name'];
$handle = fopen($file,"r");
$colsMax=0;

$matrix = array();
while (($row = fgetcsv($handle, 10000, ",")) !== FALSE) {
    $matrix[] = $row;
}
$rows = count($matrix);
$cols = count($matrix[$row]);

I then create a table in the database with the correct number of columns (with field names A1, A2 etc. which are changed later):

//create importDataTable table for all the data
$sql = "CREATE TABLE $tblImportedData (id INT(6) UNSIGNED AUTO_INCREMENT PRIMARY KEY";
for ($colloop=0; $colloop<$cols; $colloop++) {
    $sql = $sql . ", A$colloop VARCHAR(80)";
}
$sql = $sql . ");";
mysqli_query($connect, $sql);

The data is then inserted into the table row by row:

for ($rowloop=0; $rowloop < $rows; $rowloop++) { //loops through each row
    $dataToWrite = $matrix[$rowloop][0];        
    $sql = "INSERT INTO $tblImportedData SET "; 
    for ($colloop=0; $colloop < $cols; $colloop++) {  //loops through each column
        $dataToWrite = isset($matrix[$rowloop][$colloop]) ? $matrix[$rowloop][$colloop] : '';
        $sql = $sql . ", A$colloop = '" . $dataToWrite . "'";
    }
    $sql = $sql . ";";
    mysqli_query($connect, $sql);
}
8
  • 1
    WARNING: When using mysqli you should be using parameterized queries and bind_param to add any data to your query. DO NOT use string interpolation or concatenation to accomplish this because you have created a severe SQL injection bug. NEVER put $_POST, $_GET or data of any kind directly into a query, it can be very harmful if someone seeks to exploit your mistake. Commented Oct 28, 2019 at 17:20
  • Note: The object-oriented interface to mysqli is significantly less verbose, making code easier to read and audit, and is not easily confused with the obsolete mysql_query interface where missing a single i can cause trouble. Example: $db = new mysqli(…) and $db->prepare("…") The procedural interface is largely an artifact from the PHP 4 era when mysqli API was introduced and should not be used in new code. Commented Oct 28, 2019 at 17:20
  • If you can, just load it in directly using LOAD DATA INFILE. Commented Oct 28, 2019 at 17:20
  • 1
    It's often easier to do post-processing in the database than it is to do pre-processing and slowly insert the data. Import into a temporary table, rework the data, then merge it in the final location with INSERT INTO ... SELECT FROM ... Commented Oct 28, 2019 at 17:24
  • 1
    @tadman Thanks for your assistance. LOAD DATA INFILE working and much quicker. Appreciate your help. Commented Oct 29, 2019 at 23:04

1 Answer 1

2

As @tadman suggested in his comment, you should use the LOAD DATA INFILE functionality that come out of the box with mysql.

Sign up to request clarification or add additional context in comments.

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.