Generate random number after value in bash

Question

I want my machine to automatically download some files. That doesn't have to be very efficient. So I decided to do this with a bash script.

It works so far when I encode the URL hardly. But I want to get the files retrieved in irregular order and I thought I would use simple variables. How do I get the random number into my variable?

My approach

data_link0="https://example.com/target1.html"
data_link1="https://example.com/target2.html"
data_link2="https://example.com/target3.html"
data_link3="https://example.com/target4.html"

useragent0="Mozilla/5.0 (iPhone; CPU iPhone OS 10_0_1 like Mac OS X) AppleWebKit/602.1.50 (KHTML, like Gecko) Version/10.0 Mobile/14A403 Safari/602.1"
useragent1="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/604.5.6 (KHTML, like Gecko) Version/11.0.3 Safari/604.5.6"
useragent3="Mozilla/5.0 (Windows 7; ) Gecko/geckotrail Firefox/firefoxversion"

wget --user-agent="$user_agent[$((RANDOM % 3))]" "$datei_link$((RANDOM % 3))"

unfortunately does not work.

You should know that $(( RANDOM % k )) does not generate a random number uniformly from 0 to k - 1, unless k is a power of 2. (Proof: k does not divide 32768.) If it is important that your distribution be uniform, you'll need to take extra measures to effect this. — wchargin
– wchargin, Commented Mar 14, 2018 at 4:22

Gilles Quénot · Accepted Answer · 2018-03-16 00:39:58Z

As far as you need to retrieve all the urls, a better way would be using shuf (GNU/linux coreutils) (or sort -R coreutils too):

shuf file | xargs wget

File :

$ cat file
"https://example.com/target1.html"
"https://example.com/target2.html"
"https://example.com/target3.html"
"https://example.com/target4.html"

man 1 shuf

NAME

shuf - generate random permutations

New comments, new needs, new code :

(requiring random user-agent)

$ cat uas
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.52 Safari/537.36
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.52 Safari/537.36 OPR/15.0.1147.100
Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; AS; rv:11.0) like Gecko

Code :

shuf file | while read url; do
    wget --user-agent="$(shuf -n1 uas)" "$url"
done

If you prefer to keep your way (one url) :

data_link=(
    "https://example.com/target1.html"
    "https://example.com/target2.html"
    "https://example.com/target3.html"
    "https://example.com/target4.html"
)
user_agent=(
    "Mozilla/5.0 (iPhone; CPU iPhone OS 10_0_1 like Mac OS X) AppleWebKit/602.1.50 (KHTML, like Gecko) Version/10.0 Mobile/14A403 Safari/602.1"
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/604.5.6 (KHTML, like Gecko) Version/11.0.3 Safari/604.5.6"
    "Mozilla/5.0 (Windows 7; ) Gecko/geckotrail Firefox/firefoxversion"
)

wget --user-agent="${user_agent[RANDOM % ${#user_agent[@]} ]}" "${data_link[RANDOM % ${#data_link[@]}]}"

Your way for all urls and user-agent (both randomized) :

for i in $(seq 0 $((${#data_link[@]} -1)) | shuf); do
    wget -U "${user_agent[RANDOM % ${#user_agent[@]}]}" "${data_link[i]}"
done

Wow, it runs really good. But ... … now I can not fill in the option "--user-agent=" How can I fill in an randomized *user agent* from a file? — Wilhelm
– Wilhelm, Commented Mar 13, 2018 at 21:56
Something like uas=( firefox waterfox chromium opera ); shuf file | xargs wget --user-agent="${uas[ RANDOM % ${#uas[@]}]}" (adapt to your needs, this is not real user-agents) — Gilles Quénot
– Gilles Quénot, Commented Mar 13, 2018 at 22:05
Out of curiosity, is there any reason to prefer shuf over sort -R? — martin
– martin, Commented Mar 14, 2018 at 8:47
Both works, up to OP :) Added it at the beginning of the post — Gilles Quénot
– Gilles Quénot, Commented Mar 14, 2018 at 12:49

Stéphane Chazelas · Accepted Answer · 2018-03-14 08:30:58Z

4

Instead of defining separate variables for each string, define an array. Use ${ar[123]} to access element 123 of the array ar and ${#ar[@]} to get the size of the array.

data_link=(
    "https://example.com/target1.html"
    "https://example.com/target2.html"
    "https://example.com/target3.html"
    "https://example.com/target4.html"
)
user_agent=(
    "Mozilla/5.0 (iPhone; CPU iPhone OS 10_0_1 like Mac OS X) AppleWebKit/602.1.50 (KHTML, like Gecko) Version/10.0 Mobile/14A403 Safari/602.1"
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/604.5.6 (KHTML, like Gecko) Version/11.0.3 Safari/604.5.6"
    "Mozilla/5.0 (Windows 7; ) Gecko/geckotrail Firefox/firefoxversion"
)

wget --user-agent="${user_agent[RANDOM % ${#user_agent[@]}]}" "${data_link[RANDOM % ${#data_link[@]}]}"

edited Mar 14, 2018 at 8:30

Stéphane Chazelas

586k96 gold badges1.1k silver badges1.7k bronze badges

answered Mar 13, 2018 at 22:14

Gilles 'SO- stop being evil'

866k205 gold badges1.8k silver badges2.3k bronze badges

Good if OP want only to retrieve one url at a time ;) (And I get a syntax error)

Gilles Quénot
– Gilles Quénot

2018-03-13 22:17:18 +00:00
Commented Mar 13, 2018 at 22:17
And no need $(( )) : uas=( foo bar base); echo "${uas[RANDOM % ${#uas[@]}]}"

Gilles Quénot
– Gilles Quénot

2018-03-13 22:22:48 +00:00
Commented Mar 13, 2018 at 22:22
I tested it, but I get this error 'Syntax error: "(" unexpected'

Wilhelm
– Wilhelm

2018-03-13 23:51:01 +00:00
Commented Mar 13, 2018 at 23:51

Add a comment |

Stack Exchange Network

Generate random number after value in bash

2 Answers 2

File :

New comments, new needs, new code :

Code :

If you prefer to keep your way (one url) :

Your way for all urls and user-agent (both randomized) :

You must log in to answer this question.

Hot Network Questions

Generate random number after value in bash

2 Answers 2

File :

New comments, new needs, new code :

Code :

If you prefer to keep your way (one url) :

Your way for all urls and user-agent (both randomized) :

You must log in to answer this question.

Related

Hot Network Questions