UNIX shell script reading csv

Question

I have a csv file. I would like to put the fields into different variables. Supposed there are three fields in each line of the csv file. I have this code:

csvfile=test.csv
while read inline; do
   var1=`echo $inline | awk -F',' '{print $1}'`
   var2=`echo $inline | awk -F',' '{print $2}'`
   var3=`echo $inline | awk -F',' '{print $3}'`
   .
   .
   .
done < $csvfile

This code is good. However, if a field is coded with an embedded comma, then, it would not work. Any suggestion? For example:

how,are,you
I,"am,  very",good
this,is,"a, line"

The code isn't good, it contains several bugs, deprecated constructs, and is the wrong approach to doing what you're trying to do plus trying to save the values in different variables is almost certainly a bad idea anyway. If you post concise, testable sample input and expected output we can help you. — Ed Morton
– Ed Morton, Commented Dec 3, 2019 at 19:32
Sounds like a job for a dedicated tool like csvkit; short of that, you could use gawk with FPAT: gnu.org/software/gawk/manual/gawk.html#Splitting-By-Content — Benjamin W.
– Benjamin W., Commented Dec 3, 2019 at 19:33
What do you want to do with the vars you have set? For each line the value changes, so I think you want to do something with them inside the loop. — Walter A
– Walter A, Commented Dec 3, 2019 at 22:55

Strick · Accepted Answer · 2019-12-04 08:12:49Z

1

This may not be the perfect solution but it will work in your case.

[cloudera@quickstart Documents]$ cat cd.csv 
a,b,c
d,"e,f",g

File content

csvfile=cd.csv
while read inline; do


var1=`echo $inline | awk -F'"' -v OFS='' '{ for (i=2; i<=NF; i+=2) gsub(",", "*", $i) }1' | awk -F',' '{print $1}' | sed 's/*/,/g'`
var2=`echo $inline | awk -F'"' -v OFS='' '{ for (i=2; i<=NF; i+=2) gsub(",", "*", $i) }1' | awk -F',' '{print $2}' | sed 's/*/,/g'`  
var3=`echo $inline | awk -F'"' -v OFS='' '{ for (i=2; i<=NF; i+=2) gsub(",", "*", $i) }1' | awk -F',' '{print $3}' | sed 's/*/,/g'`


echo  $var1 " " $var2 " " $var3 


done< $csvfile

Output :

[cloudera@quickstart Documents]$ sh a.sh 
a   b   c
d   e,f   g

So basically first we are trying to handle "," in data and then replacing the "," with "*" to get desired column using awk and then reverting * to "," again to get actual field value

answered Dec 4, 2019 at 8:12

Strick

1,65213 silver badges17 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

pat Over a year ago

I got errors when try this script: awk: syntax error near line 1 awk: bailing out near line 1 awk: syntax error near line 1 awk: bailing out near line 1 awk: syntax error near line 1 awk: bailing out near line 1

pat Over a year ago

Is there a more efficient way to put the fields into variables? In my actual csv file, I have 15 fields on each line and have more than 8000 lines.

Strick Over a year ago

You can try to store echo $inline | awk -F'"' -v OFS='' '{ for (i=2; i<=NF; i+=2) gsub(",", "*", $i) }1' part in form of array and then use them further

Strick Over a year ago

Hi Pat if above answer solves your query can you please upvote and mark it as answer

Collectives™ on Stack Overflow

UNIX shell script reading csv

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest