python - How to catch an exception in the for loop iterator

Question

This is a for loop in Python:

for_stmt ::=  "for" target_list "in" expression_list ":" suite

Normally, when yielding a value from the expression_list raises an exception, the loop aborts. Is there an elegant way (short of rewriting the loop using while True or something similar) to catch this exception and continue the loop?

Here is an example:

import csv

csv.field_size_limit(10)

reader = csv.reader(open('test.csv', 'r'))
for line in reader:
    print(line)

with this file:

foo,bar,baz
xxx,veryverylong,yyy
abc,def,ghi

This aborts at the second line. I would like a way to skip or log the failing lines and continue.

question from:https://stackoverflow.com/questions/13653783/how-to-catch-an-exception-in-the-for-loop-iterator

Answer 1

If your inner iterable can be continued after an exception, all you need to wrap it is a trivial generator:

def wrapper(gen):
  while True:
    try:
      yield next(gen)
    except StopIteration:
      break
    except Exception as e:
      print(e) # or whatever kind of logging you want

For example:

In [9]: list(wrapper(csv.reader(open('test.csv', 'r'))))
field larger than field limit (10)
Out[9]: [['foo', 'bar', 'baz'], ['abc', 'def', 'ghi']]

On the other hand, if the inner iterator can't be continued after an exception, there's no way you can wrap it:

def raisinggenfunc():
    yield 1
    raise ValueError("spurious error")
    yield 3

In [11]: list(wrapper(raisinggenfunc()))
spurious error
Out[11]: [1]

Any generator created by calling a Python generator function or evaluating a generator expression will not be resumable.

In such a case, you need to find some way to create a new iterator that resumes iteration. For something like csv.reader, that would mean reading n lines from the file before wrapping it in a csv.reader. In other cases it might mean passing n to the constructor. In other cases—as with raisinggenfunc above, it's just not possible.