10

I have file.txt that I need to read into a Bash array. Then I need to remove spaces, double quotes and all but the first comma in every entry. Here's how far I've gotten:

$ cat file.txt
10,this
2 0 , i s
30,"all"
40,I
50,n,e,e,d,2
60",s e,e"

$ cat script.sh
#!/bin/bash
readarray -t ARRAY<$1
ARRAY=( "${ARRAY[@]// /}" )
ARRAY=( "${ARRAY[@]//\"/}" )
for ELEMENT in "${ARRAY[@]}";do
    echo "|ELEMENT|$ELEMENT|"
done

$ ./script.sh file.txt
|ELEMENT|10,this|
|ELEMENT|20,is|
|ELEMENT|30,all|
|ELEMENT|40,I|
|ELEMENT|50,n,e,e,d,2|
|ELEMENT|60,se,e|

Which works great except for the comma situation. I'm aware that there are multiple ways to skin this cat, but due to the larger script this is a part of, I'd really like to use parameter substitution to get to here:

|ELEMENT|10,this|
|ELEMENT|20,is|
|ELEMENT|30,all|
|ELEMENT|40,I|
|ELEMENT|50,need2|
|ELEMENT|60,see|

Is this possible via parameter substitution?

Improve this question
4
  • 3
    Is there any reason you need to keep the text in an array, and why you can't let e.g. awk or sed do the processing of the data? Commented Mar 26, 2019 at 15:49
  • @Jeff -- Looping over the array will be a nightmare to implement in the larger script I'm working on. Commented Mar 26, 2019 at 16:04
  • 4
    @JonRed I don't know what you are doing, so it's entirely possible that you may not have a choice in the matter, but generally, when you find yourself doing such complex string acrobatics in the shell, that's a very good indication that you should be using an actual programming language. The shell is not designed as a programming language, and while it can be used as one, it really isn't a good idea for more complex things. I strongly urge you to consider switching to perl or python or any other scripting language. Commented Mar 26, 2019 at 16:12
  • @terdon It's funny, I just got done saying almost the exact same thing to my colleague before I read this post. I basically said this is the final version of this script and that any further requirements will necessitate re-writing in Perl. So yeah, I definitely agree Commented Mar 26, 2019 at 17:37

5 Answers 5

Reset to default
11

As far as I can see, there's no need to read it into a bash array to create that output:

$ sed 's/[ "]//g; s/,/ /; s/,//g; s/ /,/; s/.*/|ELEMENT|&|/' <file
|ELEMENT|10,this|
|ELEMENT|20,is|
|ELEMENT|30,all|
|ELEMENT|40,I|
|ELEMENT|50,need2|
|ELEMENT|60,see|

The sed expression deletes spaces and double quotes, replaces the first comma with a space (there are no other spaces in the string at this point), deletes all other commas, restores the first comma, and the prepends and appends the extra data.

Alternatively, with GNU sed:

sed 's/[ "]//g; s/,//2g; s/.*/|ELEMENT|&|/' <file

(standard sed does not support the combination of 2 and g as flags to the s command).

Improve this answer
9
  • 1
    with GNU sed, you can use 's/,//2g to remove commas, starting with the 2nd Commented Mar 26, 2019 at 16:02
  • 2
    And, the last 2 s/// commands can be s/.*/|ELEMENT|&|/ but that may be more effort for sed. Commented Mar 26, 2019 at 16:04
  • 1
    @glennjackman Possibly, but it looks rather neat. Commented Mar 26, 2019 at 16:05
  • Yeah, this is part of a larger script. The array is necessary, not just for the output. Hence my interest in parameter substitution. I could loop over the array with this but that will be a nightmare to implement. Terndon provided a loop-free solution using sed that I'll likely fall back on if parameter substitution is a no-go. Commented Mar 26, 2019 at 16:06
  • If I wasn't tied to using an array, however, this would be best solution. Commented Mar 26, 2019 at 16:24
9

I would remove what you need to remove using sed before loading into the array (also note the lower case variable names, in general it is best to avoid capitalized variables in shell scripts):

#!/bin/bash
readarray -t array< <(sed 's/"//g; s/  *//g; s/,/"/; s/,//g; s/"/,/' "$1")
for element in "${array[@]}";do
    echo "|ELEMENT|$element|"
done

This produces the following output on your example file:

$ foo.sh file 
|ELEMENT|10,this|
|ELEMENT|20,is|
|ELEMENT|30,all|
|ELEMENT|40,I|
|ELEMENT|50,need2|
|ELEMENT|60,see|

If you really must use parameter substitution, try something like this:

#!/bin/bash
readarray -t array< "$1"
array=( "${array[@]// /}" )
array=( "${array[@]//\"/}" )
array=( "${array[@]/,/\"}" )
array=( "${array[@]//,/}" )
array=( "${array[@]/\"/,}" )

for element in "${array[@]}"; do
    echo "|ELEMENT|$element|"
done
Improve this answer
4
  • 1
    @JonRed I added a version with parameter substitution but it's complex, cumbersome and ugly. Doing this sort of thing in the shell is very rarely a good idea. Commented Mar 26, 2019 at 16:06
  • 1
    Note that if you've removed both spaces and double quotes, these characters becoma available to use instead of your RANDOMTEXTTHATWILLNEVERBEINTHEFILE. Commented Mar 26, 2019 at 16:08
  • 1
    @Kusalananda yeah, I just read your answer. Should have thought of that! Thanks :) Commented Mar 26, 2019 at 16:10
  • Directly answers the question, illustrates why my preferred solution isn't ideal, and provides the most viable alternative. You win, best answer. Commented Mar 26, 2019 at 16:20
9 
ELEMENT='50,n,e,e,d,2'
IFS=, read -r first rest <<<"$ELEMENT"
printf "%s,%s\n" "$first" "${rest//,/}"

50,need2

Get out of the habit of using ALLCAPS variable names. You'll eventually collide with a crucial "system" variable like PATH and break your code.

Improve this answer
3
  • Not parameter substitution. BUT, I was unaware that ALLCAPS variable names was a bad habit in Bash. You make a good point, one that a cursory googling definitely confirms. Thank you for improving my style! :) Commented Mar 26, 2019 at 16:14
  • 1
    I've answer questions where the person wrote PATH=something; ls $PATH and then wondered about the ls: command not found error. Commented Mar 26, 2019 at 16:17
  • 1
    There are nearly a hundred built-in variables that are named in all caps (click through this man page link) to see... Commented Mar 26, 2019 at 16:19
8

[This is essentially a more fully developed version of glenn jackmann's answer]

Building an associative array from the stripped key and value, using the first comma as separator:

declare -A arr
while IFS=, read -r k v; do arr["${k//[ \"]}"]="${v//[ ,\"]}"; done < file.txt
for k in "${!arr[@]}"; do 
  printf '|ELEMENT|%s,%s|\n' "$k" "${arr[$k]}"
done
|ELEMENT|20,is|
|ELEMENT|10,this|
|ELEMENT|50,need2|
|ELEMENT|40,I|
|ELEMENT|60,see|
|ELEMENT|30,all|
Improve this answer
6

You could loop over the array and use an intermediate variable:

for((i=0; i < "${#ARRAY[@]}"; i++))
do
  rest="${ARRAY[i]#*,}"
  ARRAY[i]="${ARRAY[i]%%,*}","${rest//,/}"
done

This assigns to rest the portion after the first comma; we then concatenate three pieces back into the original variable:

  • the portion before the first comma
  • a comma
  • the replacement in rest of every comma with nothing
Improve this answer
2
  • This was my first thought and is simple enough for the example but this is part of larger script where the array is massive and there's already loops and it would be a whole thing. This would definitely work but would be very cumbersome to implement in the larger project I'm working on. Commented Mar 26, 2019 at 16:26
  • 1
    Fair enough; I just tried to answer within the limitations (parameter expansion only). Commented Mar 26, 2019 at 16:27

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.