我尝试使用 Python 清理一些杂乱的 XML 文件,它做了三件事:
- 将 40%-50% 的标签名称从大写转换为小写
- 删除标签之间的 NULL
- 删除标签之间的空行
我在使用 BeautifulSoup 时这样做了,但是,由于我的一些 XML 文件超过 1GB,我遇到了内存问题。相反,我查看了一些流方法,如 xml.sax,但我并没有完全理解该方法。那么有人可以给我一些建议吗?
xml_str = """
<DATA>
<ROW>
<assmtid>1</assmtid>
<Year>1988</Year>
</ROW>
<ROW>
<assmtid>2</assmtid>
<Year>NULL</Year>
</ROW>
<ROW>
<assmtid>2</assmtid>
<Year>1990</Year>
</ROW>
</DATA>
"""
xml_str_update = re.sub(r">NULL", ">", xml_str)
soup = BeautifulSoup(xml_str_update, "lxml")
print soup.data.prettify().encode('utf-8').strip()
更新
经过一些测试并采纳 Jarrod Roberson 的建议后,以下是一种可能的解决方案。
import os
import xml.etree.cElementTree as etree
from cStringIO import StringIO
def getelements(xml_str):
context = iter(etree.iterparse(StringIO(xml_str), events=('start', 'end')))
event, root = next(context)
for event, elem in context:
if event == 'end' and elem.tag == "ROW":
elem.tag = elem.tag.lower()
elem.text = "\n\t\t"
elem.tail = "\n\t"
for child in elem:
child.tag = child.tag.lower()
if child.text == "NULL":
# if do not like self-closing tag,
# add ​, which is a zero width space
child.text = ""
if child.text == None:
child.text = ""
# print event, elem.tag
yield elem
root.clear()
with open(pth_to_output_xml, 'wb') as file:
# start root
file.write('<data>\n\t')
for page in getelements(xml_str):
file.write(etree.tostring(page, encoding='utf-8'))
# close root
file.write('</data>')
请您参考如下方法:
迭代解析
When building an in-memory tree is not desired or practical, use an iterative parsing technique that does not rely on reading the entire source file. lxml offers two approaches: Supplying a target parser class Using the iterparse method
import xml.etree.ElementTree as etree
for event, elem in etree.iterparse(xmL, events=('start', 'end', 'start-ns', 'end-ns')):
print event, elem
这是一个very complete tutorial关于如何做到这一点。
This will parse the XML file in chunks at a time and give it to you at every step of the way. start will trigger when a tag is first encountered. At this point elem will be empty except for elem.attrib that contains the properties of the tag. end will trigger when the closing tag is encountered, and everything in-between has been read.
然后在您的事件处理程序中,您只需写出遇到的已转换信息。
