Placing a numpy record array inside another record array python

Question

I have the following record array. But each element needs it's own record array describing information about that element. Right now this isn't code I am just displaying the information in a readable way. For example:

I have a collection of registers with attributes like width and offset.

List of registers

Registers     offsets    width

Register_1    1          5
Register_2    2          10
Register_3    3          15
Register_4    4          20

Each register also has a set of fields which have specific values like bit and length

Inside of register_1 we have the following fields

register_1
{
#field names      bit     length
field_1           1       10
field_2           2       20
field_3           3       30

}

Actual python code

dtype={
    'names' : ('name','offset','width'),
    'formats' : ('U20','U20','U20')}

registers= np.zeros(10,dtype)

registers[0]=('register_1','1','5')
registers[1]=('register_2','2','10')
registers[2]=('register_3','3','15')
registers[3]=('register_4','4','20')

Now I want to associate each register with their fields, the first thing that came to mind was nesting another record array in my register record array.

Pseudo python code

type={
    'names' : ('name','bit','length'),
    'formats' : ('U20','U20','U20')}

fields= np.zeros(10,type)

 dtype={
    'names' : ('name','offset','width',fields),
    'formats' : ('U20','U20','U20','U20')}

regsiters= np.zeros(10,dtype)

Any suggestions?

Thank you!

No the number of field's vary per register, but I wouldn't think that would be an issue. — GoldenEagle
– GoldenEagle, Commented Oct 29, 2015 at 21:18
What sort of indexing operations are you most likely to want to perform. For example, are you mostly going to be indexing by register or by field? — ali_m
– ali_m, Commented Oct 29, 2015 at 21:20
Well I will be printing out the data, sometimes by register name, others by the register name and all of its fields. — GoldenEagle
– GoldenEagle, Commented Oct 29, 2015 at 21:23
Given that it sounds like you're indexing your data exclusively by register, it seems to me that you're probably better off storing your "field" data in something more like a set of nested dicts. Numpy arrays are really designed for flat rather than nested data structures. Technically you could nest a recarray inside another by abusing the np.object dtype, but at that point you might as well use a plain Python list, since you lose the performance benefits of numpy arrays by using them to store arbitrary Python objects. — ali_m
– ali_m, Commented Oct 29, 2015 at 21:36

hpaulj · Accepted Answer · 2015-10-30 21:57:36Z

2

Turn your type (bad name!) into a dtype:

In [277]: dt1=np.dtype(type)
In [278]: dt1
Out[278]: dtype([('name', '<U20'), ('bit', '<U20'), ('length', '<U20')])

build a composite dtype. Notice that the format for 'fields' is (dt1,3) - i.e. a subarray consisting of 3 elements of type dt1.

In [279]: dt2=np.dtype({
    'names' : ('name','offset','width','fields'),
    'formats' : ('U20','U20','U20',(dt1,3))})
In [280]: registers=np.zeros(4,dtype=dt2)
In [281]: registers
Out[281]: 
array([('', '', '', [('', '', ''), ('', '', ''), ('', '', '')]),
       ('', '', '', [('', '', ''), ('', '', ''), ('', '', '')]),
       ('', '', '', [('', '', ''), ('', '', ''), ('', '', '')]),
       ('', '', '', [('', '', ''), ('', '', ''), ('', '', '')])], 
      dtype=[('name', '<U20'), ('offset', '<U20'), ('width', '<U20'), ('fields', [('name', '<U20'), ('bit', '<U20'), ('length', '<U20')], (3,))])

Fill in the names:

In [283]: registers['name']='register_1 register_2 register_3 register_4'.split()

and other 2 fields:

In [284]: registers['offset']=[1,2,3,4]
In [285]: registers['width']=[5,10,15,20]
In [286]: registers
Out[286]: 
array([('register_1', '1', '5', [('', '', ''), ('', '', ''), ('', '', '')]),
       ('register_2', '2', '10', [('', '', ''), ('', '', ''), ('', '', '')]),
       ('register_3', '3', '15', [('', '', ''), ('', '', ''), ('', '', '')]),
       ('register_4', '4', '20', [('', '', ''), ('', '', ''), ('', '', '')])], 
      dtype=[('name', '<U20'), ('offset', '<U20'), ('width', '<U20'), ('fields', [('name', '<U20'), ('bit', '<U20'), ('length', '<U20')], (3,))])

The fields value for registers[0] element can be filled in in the same way:

In [288]: registers[0]['fields']
Out[288]: 
array([('', '', ''), ('', '', ''), ('', '', '')], 
      dtype=[('name', '<U20'), ('bit', '<U20'), ('length', '<U20')])

Now that registers has 4 fields, you can't fill in 3 of them with:

In [289]: registers[0]=('register_1','1','5')
...
ValueError: size of tuple must match number of fields.

The tuple has include all the values needed for the 'fields' field (why did you choose confusing names like this?).

Create an array with dtype dt1, and fill it.

In [290]: regfield=np.zeros(3,dtype=dt1)
In [291]: regfield
Out[291]: 
array([('', '', ''), ('', '', ''), ('', '', '')], 
      dtype=[('name', '<U20'), ('bit', '<U20'), ('length', '<U20')])
In [292]: regfield[0]=('field1',1,10)
In [293]: regfield[1]=('field2',2,20)
In [294]: regfield[2]=('field3',3,30)

Now you can include it as a value in the registers tuple.

In [295]: registers[0]=('register_1','1','5',regfield)
In [296]: registers[0]
Out[296]: ('register_1', '1', '5', [('field1', '1', '10'), ('field2', '2', '20'), ('field3', '3', '30')])

I can replicate the fieldnames to the other registers with:

In [300]: fieldnames=registers['fields']['name'][0]
In [301]: registers['fields']['name']
Out[301]: 
array([['field1', 'field2', 'field3'],
       ['', '', ''],
       ['', '', ''],
       ['', '', '']], 
      dtype='<U20')
In [302]: registers['fields']['name']=fieldnames
In [303]: registers['fields']['name']
Out[303]: 
array([['field1', 'field2', 'field3'],
       ['field1', 'field2', 'field3'],
       ['field1', 'field2', 'field3'],
       ['field1', 'field2', 'field3']], 
      dtype='<U20')
In [304]: registers['fields']
Out[304]: 
array([[('field1', '1', '10'), ('field2', '2', '20'), ('field3', '3', '30')],
       [('field1', '', ''), ('field2', '', ''), ('field3', '', '')],
       [('field1', '', ''), ('field2', '', ''), ('field3', '', '')],
       [('field1', '', ''), ('field2', '', ''), ('field3', '', '')]], 
      dtype=[('name', '<U20'), ('bit', '<U20'), ('length', '<U20')])

New dtype:

In [333]: dt3=np.dtype({
    'names' : ('name','offset','width','fields'),
    'formats' : ('U20','U20','U20', 'O')})
In [334]: registers=np.zeros(4,dtype=dt3)
In [335]: registers
Out[335]: 
array([('', '', '', 0), ('', '', '', 0), ('', '', '', 0), ('', '', '', 0)], 
      dtype=[('name', '<U20'), ('offset', '<U20'), ('width', '<U20'), ('fields', 'O')])

Assign value to registers[0]. regfield the 3 element array that I defined earlier.

In [337]: registers[0]=('register_1', '1', '5', regfield)
In [338]: registers
Out[338]: 
array([ ('register_1', '1', '5', [('field1', '1', '10'), ('field2', '2', '20'), ('field3', '3', '30')]),
       ('', '', '', 0), 
       ('', '', '', 0), 
       ('', '', '', 0)], 
      dtype=[('name', '<U20'), ('offset', '<U20'), ('width', '<U20'), ('fields', 'O')])
...

Display is similar to what what we had before, but now each register can have a different number of fields.

We can verify that object in the fields' field ofregisters[0]is theregfield` array by looking at their ids.

In [343]: id(regfield)
Out[343]: 166843840
In [344]: id(registers[0]['fields'])
Out[344]: 166843840

edited Oct 30, 2015 at 21:57

answered Oct 29, 2015 at 22:06

hpaulj

233k14 gold badges260 silver badges392 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

GoldenEagle Over a year ago

Thanks for the response, so you are placing each of these in a specific point in the array? Can you explain the In[300]? And why start at 277?

hpaulj Over a year ago

The line numbers come from the Ipython session that I've been using for several days. Sorry, I didn't start a fresh session for your problem. :) At IN[300] I am experimenting with setting the fields of all of the registers with values that I'd set earlier for registers[0]`. No guarantee that it's the best way.

hpaulj Over a year ago

With dt2 the fields data is placed inline with the other registers fields. And all registers must have the same number of fields. Alternative this field could be dtype=object as @ali_m suggests. Then only a pointer is placed in the registers array, and the fields data is located in separate arrays elsewhere in memory.

GoldenEagle Over a year ago

Hi, just mess around with the code, this is exactly what I wanted thank you!

hpaulj Over a year ago

I added the dtype=object variant.

Collectives™ on Stack Overflow

Placing a numpy record array inside another record array python

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related