Read and write data from text file to numpy column in python

Question

I've been struggling to get something to work for the following text file format. My overall goal is to extract the value for one of the variable names throughout the entire text file. For example, I want all the values for B rows and D rows. Then put them in a normal numpy array and run calculations.

Here is what the data file looks like:

[SECTION1a]
[a] 1424457484310
[b] 5313402937
[c] 873348378938
[d] 882992596992
[e] 14957596088
[SECTION1b]
243 62 184 145 250 180 106 208 248 87 186 137 127 204 18 142 37 67 36 72 48     204 255 30 243 78 44 121 112 139 76 71 131 50 118 10 42 8 67 4 98 110 37 5 208   104 56 55 225 56 0 102 0 21 0 156 0 174 255 171 0 42 0 233 0 50 0 254 0 245 255   110 
[END SECTION1]
[SECTION2a]
[a] 1424457484310
[b] 5313402937
[c] 873348378938
[d] 882992596992
[e] 14957596088
[SECTION2b]
243 62 184 145 250 180 106 208 248 87 186 137 127 204 18 142 37 67 36 72 48   204 255 30 243 78 44 121 112 139 76 71 131 50 118 10 42 8 67 4 98 110 37 5 208 104 56 55 225 56 0 102 0 21 0 156 0 174 255 171 0 42 0 233 0 50 0 254 0 245 255 110 
[END SECTION2]

That pattern continues for N sections.

Currently I read the file and put it into two columns:

filename_load = fileopenbox(msg=None, title='Load Data File',
                        default="Z:\*",
                        filetypes=None)

col1_data = np.genfromtxt(filename_load, skip_header=1, dtype=None, 
usecols=(0,), usemask=True, invalid_raise=False)

col2_data = np.genfromtxt(filename_load, skip_header=1, dtype=None, 
usecols=(1,), usemask=True, invalid_raise=False)

I was going to then use where, to find the index of the value I wanted, then make a new array of those values:

arr_index = np.where(col1_data == '[b]')
new_array = col2_data[arr_index]

Problem with that is, I end up with arrays of two different sizes because of the weird file format so obviously the data in the array won't match up properly to the right variable name.

I have tried a few other alternatives and get stuck due to the weird text file format and how to read it into python.

Not sure if I should stay on this track an if so how to address the problem, or, try a totally different approach.

Thanks in advance!

Sorry, the data file looks terrible, each variable should be the start of the new row. Did not look that way in the preview, will try to fix. — Scott
– Scott, Commented Feb 21, 2015 at 22:07
You say that the number of [b] and [d] don't match up? Are they sometimes missing? Also, do you only want the data in the SECTIONXa blocks (ignore the row data in SECTIONXb)? — mfitzp
– mfitzp, Commented Feb 21, 2015 at 22:41
The way i did it. I end up with one array of about 600 and the other over 800 if i remember right. So the section b blocks must be messing up my plans which now seems obvious. I do not need section b blocks. On data out section a blocks. So, one approach could first take out those sections but how? — Scott
– Scott, Commented Feb 21, 2015 at 23:18

Dietrich · Accepted Answer · 2015-02-22 16:17:39Z

A possible solution sorting your data into hierachy of OrdedDict() dictionaries:

from collections import OrderedDict
import re


ss = """[SECTION1a]
[a] 1424457484310
[b] 5313402937
[c] 873348378938
[d] 882992596992
[e] 14957596088
[SECTION1b]
243 62 184 145 250 180 106 208 248 87 186 137 127 204 18 142 37 67 36 72 48     204 255 30 243 78 44 121 112 139 76 71 131 50 118 10 42 8 67 4 98 110 37 5 208   104 56 55 225 56 0 102 0 21 0 156 0 174 255 171 0 42 0 233 0 50 0 254 0 245 255   110
[END SECTION1]
[SECTION2a]
[a] 1424457484310
[b] 5313402937
[c] 873348378938
[d] 882992596992
[e] 14957596088
[SECTION2b]
243 62 184 145 250 180 106 208 248 87 186 137 127 204 18 142 37 67 36 72 48   204 255 30 243 78 44 121 112 139 76 71 131 50 118 10 42 8 67 4 98 110 37 5 208 104 56 55 225 56 0 102 0 21 0 156 0 174 255 171 0 42 0 233 0 50 0 254 0 245 255 110
[END SECTION2]"""

# regular expressions for matching SECTIONs
p1 = re.compile("^\[SECTION[0-9]+a\]")
p2 = re.compile("^\[SECTION[0-9]+b\]")
p3 = re.compile("^\[END SECTION[0-9]+\]")

def parse(ss):
    """ Make hierachial dict from string """
    ll, l_cnt = ss.splitlines(), 0
    d = OrderedDict()
    while l_cnt < len(ll): # iterate through lines
        l = ll[l_cnt].strip()
        if p1.match(l):  # new sub dict for [SECTION*a]
            dd, nn = OrderedDict(), l[1:-1]
            l_cnt += 1
            while (p2.match(ll[l_cnt].strip()) is None and
                   p3.match(ll[l_cnt].strip()) is None):
                ww = ll[l_cnt].split()
                dd[ww[0][1:-1]] = int(ww[1])
                l_cnt += 1
            d[nn] = dd
        elif p2.match(l):  # array of ints for [SECTION*b]
            d[l[1:-1]] = [int(w) for w in ll[l_cnt+1].split()]
            l_cnt += 2
        elif p3.match(l):
            l_cnt += 1
    return d

dd = parse(ss)

Note that you can get much more robust code, if you use an existing parsing tool (e.g., Parsley).

To retrieve'[c]' from all sections, do:

print("All entries for [c]: ", end="")
cc = [d['c'] for s,d in dd.items() if s.endswith('a')]
print(", ".join(["{}".format(c) for c in cc]))    
# Gives: All entries for [c]: 873348378938, 873348378938

Or you could traverse the whole dictionary:

def print_recdicts(d, tbw=0):
    """print the hierachial dict """
    for k,v in d.items():
        if type(v) is OrderedDict:
            print(" "*tbw + "* {}:".format(k))
            print_recdicts(v, tbw+2)
        else:
            print(" "*tbw + "* {}: {}".format(k,v))

print_recdicts(dd)
# Gives:
# * SECTION1a:
#   * a: 1424457484310
#   * b: 5313402937
# ...

mfitzp · Accepted Answer · 2015-02-22 19:17:27Z

The following should do it. It uses a running store (tally) to cope with missing values, then writes the state out when hitting the end marker.

import re
import numpy as np

filename = "yourfilenamehere.txt"

# [e] 14957596088
match_line_re = re.compile(r"^\[([a-z])\]\W(\d*)")

result = {
    'b':[],
    'd':[],
    }

tally_empty = dict( zip( result.keys(), [np.nan] * len(result) ) )

tally = tally_empty
with open(filename, 'r') as f:
    for line in f:
        if line.startswith('[END SECTION'):
            # Write accumulated data to the lists
            for k, v in tally.items():
                result[k].append(v)

            tally = tally_empty 

        else:
            # Map the items using regex
            m = match_line_re.search(line)
            if m:
                k, v = m.group(1), m.group(2)
                print(k,v)
                if k in tally:
                    tally[k] = v

b = np.array(result['b'])
d = np.array(result['d'])

Note, whatever keys are in the result dict definition will be in the output.

Collectives™ on Stack Overflow

Read and write data from text file to numpy column in python

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related