Skip to main content
 首页 » 编程设计

python之Google App Engine NDB 数据库损坏了吗

2025年05月04日25lovecherry

我不断收到随机错误,例如:

suspended generator _get_tasklet(context.py:329) raised ProtocolBufferDecodeError(corrupted) 

或

suspended generator put(context.py:796) raised ValueError(Expecting , delimiter: line 1 column 440 (char 440)) 

或

suspended generator put(context.py:796) raised ValueError(Invalid \escape: line 1 column 18002 (char 18002)) 

或

suspended generator _get_tasklet(context.py:329) raised ProtocolBufferDecodeError(truncated) 

几天前一切正常,我没有做任何更改。当我重新启动我的应用程序时,一切正常大约五分钟,直到我得到一个

suspended generator _get_tasklet(context.py:329) raised ProtocolBufferDecodeError(corrupted) 

在那之后,我在每个数据库放置或获取时都遇到了其他错误之一。导致错误的表和代码每次都不同。我不知道从哪里开始,因为错误每次都出现在新的地方。 这些只是常规的数据库放置和获取,例如

ndbstate = NdbStateJ.get_by_id(self.screen_name) 

或

ndbstate.put() 

Google 搜索无法为我指明任何特定方向。有任何想法吗? 错误

Expecting , delimiter: line 1 column 440 (char 440) 

可能是因为某些表中的某些字段类型是JSON。但是为什么突然呢?

所以也许我在某处没有正确转义,比如使用 r'{...}',但是如果某处有错误的条目,如果我无法查询,我该如何解决?为什么它会破坏所有查询的整个表?为什么它是随机的。每次都不是同一个查询。

这是一个表格的例子

class NdbStateJ(ndb.Model): 
    last_id = ndb.IntegerProperty() 
    last_search_id = ndb.IntegerProperty() 
    last_geo_id = ndb.IntegerProperty() 
    mytweet_num = ndb.IntegerProperty() 
    mentions_processed = ndb.JsonProperty() 
    previous_follower_responses = ndb.JsonProperty() 
    my_tweets_tweeted = ndb.JsonProperty() 
    responses_already_used = ndb.JsonProperty() 
    num_followed_by_cyborg = ndb.IntegerProperty(default=0) 
    num_did_not_follow_back = ndb.IntegerProperty(default=0) 
    language_model_vector = ndb.FloatProperty(repeated=True) 
    follow_wait_counter = ndb.IntegerProperty(default=0) 

下面是一个创建表的例子

ndbstate = NdbStateJ(id=screen_name, 
last_id = 37397357946732541, 
last_geo_id = 37397357946732541, 
last_search_id = 0, 
mytweet_num = 0, 
mentions_processed = [], 
previous_follower_responses = [], 
my_tweets_tweeted = [], 
responses_already_used= [], 
language_model_vector = []) 
ndbstate.put() 

请您参考如下方法:

这是数据库中格式错误的 JSON 导致的问题。我不知道为什么问题突然开始到处发生;也许谷歌方面发生了一些变化,或者我没有充分检查,新用户能够输入格式错误的数据。谁知道。

为了修复它,我从 https://stackoverflow.com/users/1011633/nizz 中获得灵感回应App Engine return JSON from JsonProperty , https://stackoverflow.com/users/1709587/mark-amery回应How to escape special characters in building a JSON string? , 和 https://stackoverflow.com/users/1639625/tobias-k回应How do I automatically fix an invalid JSON string? .

我将 ndb.JsonProperty() 替换为 ExtendedJsonProperty,其中扩展版本看起来类似于下面的代码。

import json 
from google.appengine.ext import ndb  
import logging 
logging.getLogger().setLevel(logging.DEBUG) 
import re 
 
class ExtendedJsonProperty(ndb.BlobProperty): 
    # Inspired by https://stackoverflow.com/questions/18576556/app-engine-return-json-from-jsonproperty 
    def _to_base_type(self, value): 
        logging.debug('Dumping value '+str(value)) 
        try: 
            return json.dumps(value)  
        except Exception as e: 
            logging.warning(('trying to fix error dumping from database: ') +str(e)) 
            return fix_json(value,json.dumps) 
 
    def _from_base_type(self, value): 
        # originally return json.loads(value) 
        logging.debug('Loading value '+str(value)) 
        try: 
            return json.loads(value) 
        except Exception as e: 
            logging.warning(('trying to fix error loading from database: ') +str(e)) 
            return fix_json(value,json.loads)         
 
def fix_json(s,json_fun): 
    for _i in range(len(s)): 
        try: 
            result = json_fun(s)   # try to parse... 
            return result                     
        except Exception as e:   
            logging.debug('Exception for json loads: '+str(e))           
            if 'delimiter' in str(e): 
                # E.g.: "Expecting , delimiter: line 34 column 54 (char 1158)" 
                logging.debug('Escaping quote to fix.') 
                s = escape_quote(s,e) 
            elif 'escape' in str(e): 
                # E.g.: "Invalid \escape: line 1 column 9 (char 9)" 
                logging.debug('Removing invalid escape to fix.') 
                s = remove_invalid_escape(s) 
            else: 
                break 
    return json_fun('{}') 
 
def remove_invalid_escape(value): 
    # Inspired by https://stackoverflow.com/questions/19176024/how-to-escape-special-characters-in-building-a-json-string 
    return re.sub(r'\\(?!["\\/bfnrt])', '', value) 
 
def escape_quote(s,e): 
    # Inspired by https://stackoverflow.com/questions/18514910/how-do-i-automatically-fix-an-invalid-json-string 
    # "Expecting , delimiter: line 34 column 54 (char 1158)" 
    # position of unexpected character after '"' 
    unexp = int(re.findall(r'\(char (\d+)\)', str(e))[0]) 
    # position of unescaped '"' before that 
    unesc = s.rfind(r'"', 0, unexp) 
    s = s[:unesc] + r'\"' + s[unesc+1:] 
    # position of corresponding closing '"' (+2 for inserted '\') 
    closg = s.find(r'"', unesc + 2) 
    if closg + 2 < len(s): 
        print closg, len(s) 
        s = s[:closg] + r'\"' + s[closg+1:] 
    return s