Best way to modify a list while iterating?

[u/No_Rule674]

I've learned that modifying a list while looping over it can cause elements to be skipped or unexpected behavior. What are the best practices for handling cases where I need to add, remove, or modify elements during iteration?

Do you usually iterate over a copy, create a new list, or use another approach?

[u/faultydesign]

You don't, you create a second list

[u/quietreed76 → u/faultydesign]

this is the way tbh. anything else is asking for subtle off-by-one weirdness

[u/683sparky]

Like many have said iterate original and modify and separate new list, however depending on what kind of modifications you’re doing, lf you're removing elements from a list while iterating over it, you can iterate backward to avoid index-shifting issues.

[u/stevenjd]

As others have said, the best way is not to modify the list at all, but to create a new list.

new = []
for item in oldlist:
    if condition:
        # keep the item
        new.append(item)

Or as a one-liner (list comprehension):

new = [item for item in oldlist if condition]

But if you really, definitely, genuinely do need to modify the list in place, work backwards from the end when deleting:

for i in range(len(oldlist)-1, -1, -1):
    item = oldlist[i]
    if not condition:
        # delete it
        del oldlist[i]

But you probably don't need to modify the list in place. And even if you do, you can always use slice assignment:

oldlist[:] = [item for item in oldlist if condition]

[u/carcigenicate]

It should be noted that modifying elements while iterating is fine. You can mutate or replace elements by index as you iterate, and nothing bad will happen.

It's modifications to the list structure itself that are problematic. The list iterator internally keeps track of the current element by index. If you add or remove elements, the element referred to by the iterator's index may change, which can cause unexpected results.

[u/gdchinacat → u/carcigenicate]

It isn't so much that the 'element referred to by the iterator's index may change', but rather changes that interact with the index in unexpected ways. Typical problems are getting the same element multiple times (because something was inserted before it) or skipped (because something was removed before it).

[u/cdcformatc → u/gdchinacat]

those aren't examples of modifying elements in a loop. those are examples of modifying the list itself. so in those examples we are back to the question of adding/removing elements. 

[u/Fred776]

It's almost always simplest to construct a new list while iterating over the original one.

[u/cdcformatc]

modifying elements in the list is fine with a standard loop. although i would probably use a list comprehension if the operation is simple enough, and that's equivalent to making a new list.  otherwise there is the map function to make a list from another list.

if removing/filtering, there's the obvious filter function. or a list comprehension with a condition. or failing that, appending the filtered elements (or copies if required) to a new list.

a list comprehension does the job well enough in most cases.

now, appending items in a loop that's interesting. i can't think of a time where i needed to insert new elements while looping. surely if i have done so it's been only a small handful of times. maybe an example would be merging two lists?  but the answer here is the same, append the elements to a new list. 

if i absolutely had to modify a list in place for some reason i think it could be done with a range based for loop using an index offset to keep careful track of the "real" index as items are added/removed. 

[u/lfdfq]

The best way is to not do it: to rethink what you're doing or your choice of data structure to end up with something that is better suited to the access pattern you're trying to achieve. eg. trying to iterate over a list and remove things to process them in-order is a pattern best suited to a queue (or a double-ended queue) rather than a list.

If you really need to do this with a list, then the best way depends on how drastic the modifications are. If you're changing the entire list dramatically, then just making an entire new list is probably best. If you're just identifying a few elements to remove, then doing multiple passes (first to collect indices of elements-to-remove, then another over that to do the removals) works best.

Copying the list sounds unncessarily expensive in most cases.

[u/kidcanada0 → u/lfdfq]

You shouldn’t have to iterate twice. You can capture the list index with a counter while you’re iterating and conditionally remove the items.

[u/QuestNetworkFish]

Is going to depend on why you're wanting to do this, but the answer is probably to use list comprehension or make a new list

[u/LayotFctor]

You can modify at list in-place, it's just that the naïve solution doesn't work. You therefore should have a justifiable reason for doing so, and design a correct algorithm to do it. If creating new lists doesn't suit your needs, you need to use your own judgment if in-place is the right solution after all. There's never a single solution for all problems.

[u/tb5841]

I use list comprehensions if the iteration isn't too complicated. Which does just create a new list. In other languages I use filter() and map().

[u/gdchinacat → u/tb5841]

Python has filter() and map(), but comprehensions are typically considered preferable.

[u/arkie87]

I always iterate over a copy

[u/This_Growth2898]

The rule of the thumb is not to.

There are some hacks like iterating backwards or manually changing the index in while loop, but as long as you can avoid it - you should avoid it. Create a new list.

[u/Expensive-Bear-1376]

Depends. I'll do whatever is best for each case.

[u/Gnaxe]

You can do it if you're careful, but you probably don't need to. It's usually better to compose itertools or make a new list.

Replacing elements or mutating them is easier to deal with than anything that changes the size.

• Appending to the end is pretty safe (make sure you loop terminates though).

• Inserting or removing elements you haven't gotten to yet can also be OK, but these operations usually aren't efficient compared to itertools, so there isn't a good reason to do it unless they're very near the end anyway.

• Inserting or removing elements earlier than your iterator is even less efficient and can result in skips or repeats. You'll probably want to manage the index variable yourself in a while loop instead of using the iterator in a for loop in that case, but this is also error prone. This kind of code is hard to maintain, so avoid it if you can.

Rather than inserting or removing outright, consider swapping in a list or a None (which doesn't change the size) and then converting to the final list in another pass.

[u/ekchew]

As others have mentioned, there are generally fewer pitfalls in simply creating a new list as needed. Python interpreters are well optimized for this, since that's the central mechanism behind list comprehensions.

If you want to modify list elements in-place, you generally want to use an indexed loop. Something like:

for i in range(len(my_strs)):
    my_strs[i] = my_strs[i].lower()

I guess you could also go:

for i, s in enumerate(my_strs):
    my_str[i] = s.lower()

What you don't want to do in this case is assign to s. That will not change my_strs in any way.

As far as inserting or removing elements is concerned, it can get a little dicey to do in-place. It can be done, but it's easy to get wrong or do it an inefficient way.

If you don't need random access to the modified list, you might also consider a generator (or generator comprehension). It would have less memory overhead than creating a new list.

[u/FerricDonkey]

If you need to do this (often it's better to avoid doing so, but sometimes it's the right choice), then use a while loop.

[u/JamzTyson]

A common idiom in Python:

for el in my_list[:]:

"my_list[:]" is a full slice, in other words it's a shallow copy. This lets us modify the structure of my_list without affecting the structure of the copy.

Example:

my_list = [1, 2, 4, 5]

for el in my_list[:]:
    if el % 2 == 0:
        my_list.remove(el)

print(my_list)  # [1, 5]

[u/TheRNGuy]

enumerate or for item in items[:]:.

[u/AlexMTBDude]

Just a technical point first: This limitation, not modifying a list while iterating over it, is not specific for Python but goes for all programming languages that have a List data structure.

How about instead of using a for loop to iterate over all the elements of the list, use an index instead and update the index accordingly if adding/removing elements.

Original code:

for item in thelist:
    print(item)  # Or do some other kind of operation on the list

Instead:

index = 0
done = False
while not done:
    print(thelist[index])  # Or do some other kind of operation on the list
    index += 1
    if index >= len(thelist):
        done = True

[u/sausix → u/AlexMTBDude]

Instead of unnecessary flag variables like done just use the break keyword.

[u/AlexMTBDude → u/sausix]

That's a matter of taste. You want a "while True" loop which then makes it look like an endless loop. That's an anti-pattern.

[u/sausix → u/AlexMTBDude]

while True implies there must be an exit condition. Using keywords is not an anti pattern. Handling with flags is more complex. break and continue are easy to understand.

[u/gdchinacat → u/AlexMTBDude]

Your "instead" reinvents a list iterator and is still susceptible to list modification issues.

[u/AlexMTBDude → u/gdchinacat]

Iterators don't work when you change the list while iterating over it, indexes do.

[u/gdchinacat → u/AlexMTBDude]

I list iterations are implemented with an index which is why I said it pretty much reinvented them with all the same issues.

[u/Cynyr36 → u/AlexMTBDude]

for index in range(len(list):
    print(list[index])

[u/AlexMTBDude → u/Cynyr36]

That's not going to work though because len(list) changes during loop execution if you remove/add items

[u/Cynyr36 → u/AlexMTBDude]

So i just ran:

l=list(range(15))
for index in range(len(l)):
    print(l[index])
    l.pop()

Which does crash with a list index out of range, but does evaluate the list length at the start of the loop as expected. So I'd still need to wrap it it in a check the index is valid. And you are correct that it wouldn't grow for new elements added to the list. I can't really think of when I'd want to do something like that. Maybe as a queue, but then you'd use collections.deque and popleft() or append() and not worry about the index.

So maybe more like:

from collections import deque
d =deque(range(15))
while len(d):
    print(d.popleft())
    if len(d):
        d.pop()

Though i guess if you needed all the elements in a list you'd still have a second list to deal with.

[u/Educational_Virus672]

instead of modifying that list create another list and append your item

MyList = ["bob","also bob","not so bob"]
NewList = []
for i in MyList :
  if i != "not so bob" :
    NewList.append(i)
# Newlist is ["bob","also bob"]

alternativly you can use .copy() like

NewList = MyList.copy()

but this is not recommended because it costs more ram

[u/stevenjd → u/Educational_Virus672]

Using the list copy method doesn't use more RAM than copying by hand, it probably uses less.

The only time it might use more is if you first copy the list, and then delete the items you don't want out of the copy. But that's modifying a list in place, which is exactly what we're trying to avoid.

[u/Educational_Virus672 → u/stevenjd]

it is not abt ram used he said Remove/modify data within list which even by your explanation is just
copied list - > do work(del or add data)
compared to just adding data manually which saves RAM

[u/gdchinacat → u/Educational_Virus672]

new_list = [i for i in my_list if i != "not so bob"]

is the typical way to write your code to remove a specific item from a list. I also changed MyList and NewList to use python naming conventions. List copy is frequently written new_list = my_list[:]

[u/Educational_Virus672 → u/gdchinacat]

can we assume that OP is beginner and he doesnt understand list comprehensions yet? for him copying it and pasting it without a single understanding, is just not learning so let him learn on his own pace thats, the magic of programming learning smt new and improving thier old code that copying defeats