Ad

Read Multiple Gzip Files To 1 Fileobject In Python

- 1 answer

i want to read multiple gzip file to 1 file object currently i am doing

import gzip 
a = gzip.open(path2zipfile1) 
for line in a.readline()
  #do some stuff

but i need to read from say 2 files

a = gzip.open(path2zipfile1)  #read zip1
a = gzip.open(path2zipfile2, 'rU') #appending file object with contents of 2nd file 
for line in a.readlines()
  #this should give me contents from zip1 then zip2

unable to find the right mode to do so

Ad

Answer

use itertools.chain:

import itertools, gzip

files = ['path2zipfile1', 'path2zipfile2']
it = (gzip.open(f, 'rt') for f in files)

for line in itertools.chain.from_iterable(it):
    print(line)

another version without itertools:

def gen(files):
    for f in files:
        fo = gzip.open(f, 'rt')
        while True:
            line = fo.readline()
            if not line:
                break
            yield line

files = ['path2zipfile1', 'path2zipfile2']
for line in gen(files):
    print(line)
Ad
source: stackoverflow.com
Ad