Skip to main content
 首页 » 编程设计

python之性能改进之使用 Get 方法循环

2025年04月02日14think

我已经构建了一个程序来填充数据库,到那时,它正在运行。基本上,该程序向我正在使用的应用程序发出请求(通过 REST API)返回我想要的数据,然后处理为数据库可接受的形式。

问题是:GET 方法使算法太慢,因为我正在访问特定条目的详细信息,所以对于每个条目我必须发出 1 个请求。我有接近 15000 个请求要做,银行中的每一行都需要 1 秒才能完成。

有什么方法可以使这个请求更快?我怎样才能提高这种方法的性能?顺便问一下,有什么衡量代码性能的技巧吗?

提前致谢!

代码如下:

# Retrieving all the IDs I want to get the detailed info 
abc_ids = serializers.serialize('json', modelExample.objects.all(), fields=('id')) 
abc_ids = json.loads(abc_ids) 
abc_ids_size = len(abc_ids) 
 
# Had to declare this guys right here because in the end of the code I use them in the functions to create and uptade the back 
# And python was complaining that I stated before assign. Picked random values for them. 
age = 0 
time_to_won = 0 
data = '2016-01-01 00:00:00' 
 
# First Loop -> Request to the detailed info of ABC 
for x in range(0, abc_ids_size): 
 
id = abc_ids[x]['fields']['id'] 
url = requests.get( 
    'https://api.example.com/v3/abc/' + str( 
        id) + '?api_token=123123123') 
 
info = info.json() 
dealx = dict(info) 
 
# Second Loop -> Picking the info I want to uptade and create in the bank 
for key, result in dealx['data'].items(): 
    # Relevant only for ModelExample -> UPTADE 
    if key == 'age': 
        result = dict(result) 
        age = result['total_seconds'] 
    # Relevant only For ModelExample -> UPTADE 
    elif key == 'average_time_to_won': 
        result = dict(result) 
        time_to_won = result['total_seconds'] 
 
    # Relevant For Model_Example2 -> CREATE 
    # Storing a date here to use up foward in a datetime manipulation 
    if key == 'add_time': 
        data = str(result) 
 
    elif key == 'time_stage': 
 
        # Each stage has a total of seconds that the user stayed in. 
        y = result['times_in_stages'] 
        # The user can be in any stage he want, there's no rule about the order. 
        # But there's a record of the order he chose. 
        z = result['order_of_stages'] 
 
        # Creating a list to fill up with all stages info and use in the bulk_create. 
        data_set = [] 
        index = 0 
 
        # Setting the number of repititions base on the number of the stages in the list. 
        for elemento in range(0, len(z)): 
            data_set_i = {} 
            # The index is to define the order of the stages. 
            index = index + 1 
 
            for key_1, result_1 in y.items(): 
                if int(key_1) == z[elemento]: 
                    data_set_i['stage_id'] = int(z[elemento]) 
                    data_set_i['index'] = int(index) 
                    data_set_i['abc_id'] = id 
 
                    # Datetime manipulation 
                    if result_1 == 0 and index == 1: 
                        data_set_i['add_date'] = data 
 
                    # I know that I totally repeated the code here, I was trying to get this part shorter 
                    # But I could not get it right. 
                    elif result_1 > 0 and index == 1: 
                        data_t = datetime.strptime(data, "%Y-%m-%d %H:%M:%S") 
                        data_sum = data_t + timedelta(seconds=result_1) 
                        data_sum += timedelta(seconds=3) 
                        data_nova = str(data_sum.year) + '-' + str(formaters.DateNine( 
                            data_sum.month)) + '-' + str(formaters.DateNine(data_sum.day)) + ' ' + str( 
                            data_sum.hour) + ':' + str(formaters.DateNine(data_sum.minute)) + ':' + str( 
                            formaters.DateNine(data_sum.second)) 
                        data_set_i['add_date'] = str(data_nova) 
 
                    else: 
                        data_t = datetime.strptime(data_set[elemento - 1]['add_date'], "%Y-%m-%d %H:%M:%S") 
                        data_sum = data_t + timedelta(seconds=result_1) 
                        data_sum += timedelta(seconds=3) 
                        data_nova = str(data_sum.year) + '-' + str(formaters.DateNine( 
                            data_sum.month)) + '-' + str(formaters.DateNine(data_sum.day)) + ' ' + str( 
                            data_sum.hour) + ':' + str(formaters.DateNine(data_sum.minute)) + ':' + str( 
                            formaters.DateNine(data_sum.second)) 
                        data_set_i['add_date'] = str(data_nova) 
 
                    data_set.append(data_set_i) 
 
Model_Example2_List = [Model_Example2(**vals) for vals in data_set] 
Model_Example2.objects.bulk_create(Model_Example2_List) 
 
ModelExample.objects.filter(abc_id=id).update(age=age, time_to_won=time_to_won) 

请您参考如下方法:

如果瓶颈在您的网络请求中,除了使用 gzip 或 deflate 但使用 requests 之外,您无能为力。 ..

The gzip and deflate transfer-encodings are automatically decoded for you.

如果想加倍确定,可以在get请求中加入如下headers。

{ 'Accept-Encoding': 'gzip,deflate'} 

另一种选择是使用线程并让许多请求并行运行,如果您有大量带宽和多个内核,这是一个不错的选择。

最后,有很多不同的方法来分析 python,包括 cprofile + kcachegrind组合。