Fastest way to iterate over a large list containing strings in Python?

Question

I have a large list that contains 30000 other lists. The large list contains a list of a string and two other lists. Here is an example:

large_list = [
    ...
    ['maj', [4, 7], ['3', '5']],
    ['maj7', [4, 7, 11], ['3', '5', '7']],
    ...
    ]

I want to iterate through this large_list and find the smaller list by looking for the name (maj, maj7, etc.).

What is the fastest way of doing this?

I had thought to put the list in a .txt file and reading line by line, but I don't really know if that's faster. All I know is it takes less of a toll on my IDE, as having a list of 30000 lines makes any coding sluggish.

where are the 3000000 lines coming from? how do they get into python? — Joran Beasley
– Joran Beasley, Commented May 14, 2020 at 5:24
you might want to look at the solution implemented here stackoverflow.com/a/39923218/12337794 — Bermylle Razon
– Bermylle Razon, Commented May 14, 2020 at 5:29
Would it be more efficient to declare/convert your structure as/to a dictionary, so you can do large_dict['maj7'] to do the lookup. BTW, your syntax looks wrong - should there be a close square bracket at the end of the two sample lines? — Ken Y-N
– Ken Y-N, Commented May 14, 2020 at 5:29
The lines came from other code that generated the correct attributes @Joran — RangeError
– RangeError, Commented May 14, 2020 at 6:00

Joran Beasley · Accepted Answer · 2020-05-14 19:44:47Z

1

import pandas
# convert it to a dataframe
df = pandas.DataFrame(largeList)
#select all the rows where the zeroth element starts with "maj"
mask = df[0].str.startswith("maj")
print(df[mask])

edited May 14, 2020 at 19:44

answered May 14, 2020 at 5:25

Joran Beasley

114k13 gold badges167 silver badges187 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

RangeError Over a year ago

I hadn't heard of DataFrame, but this looks promising. Thanks!

gehbiszumeis Over a year ago

Please put your answer always in context instead of just pasting code. See here for more details.

Alvison Hunter · Accepted Answer · 2020-05-14 05:59:06Z

0

Hello Bud,

I am pretty new to Python, however, in this case, my best guess to find the smallest of them all sublist in this gigantic monster called large_list is by using the code shared below:

large_list=[['maj', 4, 7], ['3', '5'], ['maj9', 7], ['maj3', 100], ['maj4', 1360], ['maj7', 4, 7, 11], ['3', '5', '7']]
def getTheMinGuy(mylist):
    minimum = min(mylist)
    return print(f"The smaller sublist on this list is => {minimum}")
getTheMinGuy(large_list)

I certainly hope this helps finding the best solution for this issue, champion. Cheers!

answered May 14, 2020 at 5:59

Alvison Hunter

6835 silver badges13 bronze badges

1 Comment

Joran Beasley Over a year ago

how does this help him solve the problem? i dont think this code will do what you think it will do... he wants a sublist containing all the elements that start with maj or some variant of maj

Collectives™ on Stack Overflow

Fastest way to iterate over a large list containing strings in Python?

2 Answers 2

2 Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related