how to merge multiple lists in python

Question

I have multiple lists like below

[u'a', 11, u'P']
[u'a', 11, u'A']
[u'b', 2, u'P']
[u'c', 1, u'P']
[u'c', 2, u'P']
[u'd', 1, u'P']
[u'e', 3, u'P']
[u'f', 2, u'P']
[u'a', 1, u'P']
[u'a', 2, u'P']
[u'b', 1, u'P']
[u'b', 11, u'P']

how to merge above lists,to loop the list to add up like below

[u'a', 11, u'P'] + [u'a', 2, u'P'] + [u'a', 11, u'A'] = ['a',('P' : 13) ,('A': 11)]

[u'b', 2, u'P'] + [u'b', 1, u'P'] + [u'b', 11, u'P'] = ['b',14,p]

output should like below :

['a',('P' : 13) ,('A': 11)]
['b',14,'p']

Senthil Kumaran · Accepted Answer · 2011-03-03 06:42:52Z

2

You could consider using collections.defaultdict and iterating through the list of dicts in the values.

import collections
d = collections.defaultdict(list)
l = [[u'a', 11, u'P'],[u'a', 11, u'A'],[u'a', 3, u'P'],[u'b', 2, u'P'],[u'c', 1, u'P'],[u'c', 2, u'P'],[u'd', 1, u'P'],[u'e', 3, u'P']]
for k1, v, k2 in l:
    if k1 in d:
            d[k1].append({k2:v})
    else: 
        d[k1] = [{k2:v}]

newdict = {}
for key,value in d.items():
    newvalue = {}
    for valuedict in value:
        for key2,value2 in valuedict.items():
            if key2 in newvalue:
                newvalue[key2] += value2
            else:
                newvalue[key2] = value2
    newdict[key] = newvalue

print newdict

And this would give you

{u'a': {u'A': 11, u'P': 14}, u'c': {u'P': 3}, u'b': {u'P': 2}, u'e': {u'P': 3}, u'd': {u'P': 1}}

answered Mar 3, 2011 at 6:42

Senthil Kumaran

57.3k15 gold badges99 silver badges139 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

sush Over a year ago

this also works,thanks a ton senthil,here i have a question,here i'll get a put as individual list for example [u'a', 11, u'P'] [u'a', 11, u'A'],how to make them set of lists like [[u'a', 11, u'P'], [u'a', 11, u'A']]

Senthil Kumaran Over a year ago

@Sushanth - It depends upon how are you getting the original items. Create an empty list l = [] and for each item you get do a l.append(item). If it is a '\n' seperated string, you could do a "item1\nitem2\nitem3".split('\n') to get that list too.

sush Over a year ago

its throws error >>> for ll in mm : ... l.append(ll.values()) ... Traceback (most recent call last): File "<console>", line 2, in <module> AttributeError: 'dict' object has no attribute 'append'

Senthil Kumaran Over a year ago

Traceback says l is a dict. You should use l = []

Community · Accepted Answer · 2017-05-23 11:53:01Z

1

The output that you want looks slightly strange due to the inconsistency between the two cases. You could trivially change this example to get whatever output you want, however:

lists = [
 [u'a', 11, u'P'],
 [u'a', 11, u'A'],
 [u'b', 2, u'P'],
 [u'c', 1, u'P'],
 [u'c', 2, u'P'],
 [u'd', 1, u'P'],
 [u'e', 3, u'P'],
 [u'f', 2, u'P'],
 [u'a', 1, u'P'],
 [u'a', 2, u'P'],
 [u'b', 1, u'P'],
 [u'b', 11, u'P']]

# Each key in this dictionary will be one of the first elements
# from the lists shown above.  The values will be dictionaries
# mapping a letter (one of the third elements in each list) to
# their total count (i.e. the sum of the second elements matching
# the other two columns)
from collections import defaultdict
results = defaultdict(dict)

for main_key, count, subkey in lists:
    d = results[main_key]
    d[subkey] = d.get(subkey,0) + count

for main_key, values in results.items():
    print main_key, "=>", values

The output is:

a => {u'A': 11, u'P': 14}
c => {u'P': 3}
b => {u'P': 14}
e => {u'P': 3}
d => {u'P': 1}
f => {u'P': 2}

Update: Thanks to sharjeel for suggesting in the comment below that I remove the setdefault by using defaultdict instead.

Update 2: In your further question in the comments below, you indicate that instead you want as output "[a] set of lists like [[u'a', 11, u'P'], [u'a', 11, u'A']". (I'm assuming for the moment that you mean a list of lists rather than a set, but that's almost as easy.) In order to construct such a list of lists , you could replace the loop that prints the values with the following:

lists_output = []

for main_key, values in results.items():
    for subkey, count in values.items():
       lists_output.append([main_key,count,subkey])

print lists_output

... which will give the output:

[[u'a', 11, u'A'], [u'a', 14, u'P'], [u'c', 3, u'P'], [u'b', 14, u'P'], [u'e', 3, u'P'],
 [u'd', 1, u'P'], [u'f', 2, u'P']]

edited May 23, 2017 at 11:53

CommunityBot

11 silver badge

answered Mar 3, 2011 at 6:14

Mark Longair

472k77 gold badges427 silver badges330 bronze badges

8 Comments

Senthil Kumaran Over a year ago

Okay, you already got your solution. I have posted an alternative solution too.

sharjeel Over a year ago

Instead of setdefault, you can use defaultdict from collections:

Mark Longair Over a year ago

@sharjeel: thanks for that suggestion - I've changed my answer and added a note of thanks.

sush Over a year ago

@mark,i have a question,here i'll get a result as individual list for example [u'a', 11, u'P'] [u'a', 11, u'A'],how to make them set of lists like [[u'a', 11, u'P'], [u'a', 11, u'A']]

Mark Longair Over a year ago

@sushanth: That's not the output you'll get - you get a dictionary of dictionaries using the example I gave. However, I've updated my answer at the end to explain how to produce output like that you describe.

|

sharjeel · Accepted Answer · 2011-03-03 06:55:12Z

Its a one liner solution if you use groupby from itertools.

Put all the lists in one list say lst.

lst = [
    [u'a', 11, u'P']
    [u'a', 11, u'A']
    [u'b', 2, u'P']
    [u'c', 1, u'P']
    [u'c', 2, u'P']
    [u'd', 1, u'P']
    [u'e', 3, u'P']
    [u'f', 2, u'P']
    [u'a', 1, u'P']
    [u'a', 2, u'P']
    [u'b', 1, u'P']
    [u'b', 11, u'P']
]

Now use group by in outer elements to group over a,b,c etc. then within each grouped data group over the third element i.e. P,A etc. The further grouped data needs to be summed.

Here is the solution:

from itertools import groupby
result = dict(
                ( k, dict( (k1, sum([i[1] for i in g2])) for k1, g2 in groupby(g, key=lambda y: y[2] ) ) )
                for k, g in groupby(lst, key=lambda x: x[0])
            )

To have a better understanding, I would recommend you to play around with single groupby then jump to nested groupby.

Here are a few links:

http://docs.python.org/library/itertools.html#itertools.groupby

http://www.builderau.com.au/program/python/soa/Python-groupby-the-iterator-swiss-army-knife/0,2000064084,339280431,00.htm

Collectives™ on Stack Overflow

how to merge multiple lists in python

3 Answers 3

4 Comments

8 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

4 Comments

8 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related