最新消息:20210816 当前crifan.com域名已被污染,为防止失联,请关注(页面右下角的)公众号

【已解决】Antd Pro中前端列表页面loading加载很慢

加载 crifan 240浏览 0评论
之前别人做的一个内容管理系统,后台是Django,前端是reactjs的Antd Pro。
之前一直发现前端页面loading很慢:
之前一直不知道原因,后来的后来发现是:
后端代码竟然是:
对于访问接口:
http://localhost:65000/api/v1/scripts/?
由于没有检测到page的参数,竟然返回所有的page的数据
-》而此处数据量很大(有几百几千个)
-〉所以导致后台返回和前端加载,都要耗时,所以loading时间很长
-》所以现在需要去优化:
而经过调试发现:
此处虽然后台接口,传入page=1但是返回所以数据是有问题
但是更加有问题的是前端页面,前端页面已经返回了数据:
-》
传递到redux的store,更新了props中的scripts了:
但是等待了半天之后,最后才调用此处的:
script fetch queryScripts: response=
-》即:
此处yield后的数据已经返回了
render都已经显示了
但是loading还存在,所以去:
【已解决】Antd Pro中如何绑定loading确保返回后正在加载立刻消失
后来感觉像是:
const response = yield call(queryScripts, payload);
返回的太慢,然后如果返回了,则之后的queryList和endLoading都是正常快速执行的。
-》所以要去找到为何:
yield call(queryScripts, payload);
返回太慢的原因
ant design pro yield call 太慢
antd pro yield call 太慢
antd pro yield call 返回 很慢
dispatch 怎么获取返回信息 · Issue #1540 · ant-design/ant-design-pro
还是去搞清楚,此处的call的源码 和具体的调用路径
然后发现了,好像没什么特殊的,但是就是很慢:
中间差了8秒
而实际上,在此之前,早就获取到promise的返回的数据了:
queryScripts: params=
1. {page: 1, page_size: 20}
16:40:18.297request.js:90 request: url=http://localhost:65000/api/v1/scripts/?page=1&page_size=20, options=
1. {headers: {…}}
, okOrErrorCallback= undefined
16:40:18.300api.js:216 queryScriptsResp=
1. Promise {<pending>}
    1. __proto__:Promise
    2. [[PromiseStatus]]:"resolved"
    3. [[PromiseValue]]:Object
不过看到此处三个Fetch同时返回的感觉:
[Violation] 'click' handler took 526ms
16:40:18.788 [Violation] Forced reflow while executing JavaScript took 71ms
16:40:18.827 request.js:113 Fetch finished loading: OPTIONS "
http://localhost:65000/api/v1/scripts/?page=1&page_size=20
".
request @ request.js:113
_callee33$ @ api.js:213
tryCatch @ runtime.js:62
...
updateScriptList @ AllScriptList.js:270
componentWillMount @ AllScriptList.js:61
callComponentWillMount @ react-dom.development.js:11507
...
dispatchInteractiveEvent @ react-dom.development.js:4532
16:40:18.857 request.js:113 Fetch finished loading: OPTIONS "
http://localhost:65000/api/v1/topics/?page_size=1000
".
request @ request.js:113
_callee32$ @ api.js:204
tryCatch @ runtime.js:62
...
dispatchInteractiveEvent @ react-dom.development.js:4532
16:40:18.866 request.js:113 Fetch finished loading: OPTIONS "
http://localhost:65000/api/v1/function_groups/?function=1
".
感觉像是浏览器的js的Fetch的promise返回很慢:
【已解决】Chrome中js的fetch response很慢
期间:
【已解决】Django中如何自定义返回分页数据
此处继续调试发现一个现象:
js中有返回的时间,相差了8秒
分别是21秒的:Fetch finished loading: OPTIONS “http://localhost:65000/api/v1/scripts/?page=1&page_size=20“.
和29秒的fetch response=
而此处Django后台的log是:
127.0.0.1 - - [10/Aug/2018 22:50:20] "OPTIONS /api/v1/scripts/?page=1&page_size=20 HTTP/1.1" 200 -
127.0.0.1 - - [10/Aug/2018 22:50:29] "GET /api/v1/scripts/?page=1&page_size=20 HTTP/1.1" 200 -
难道是:
此处OPTIONS之后再GET,就一定会需要间隔这么长时间?
django jwt api OPTIONS too slow
django DWF api OPTIONS too slow
Throttling – Django REST framework
搜了下,项目中没有DEFAULT_THROTTLE_CLASSES
django DWF GET too slow
django DWF some api too slow
Django (?) really slow with large datasets after doing some python profiling – Stack Overflow
后来再去调试发现:
貌似就是现有之前别人写的代码的逻辑,耗时太长:
这中间的9秒左右的耗费时间分别是:
(1)检索history占了4秒 到了:23:03:04
        for curHisotryIdIdx, eachHistoryId in enumerate(historyIdList):
            logger.info("===[%d] eachHistoryId=%s", curHisotryIdIdx, eachHistoryId)
            history = History.objects.get(pk=eachHistoryId)
            logger.info("history=%s", history)
            orderedScriptAllHistory = history.script_history.all().order_by('version')
            logger.info("orderedScriptAllHistory=%s", orderedScriptAllHistory)
            lastHistory = orderedScriptAllHistory.last()
            logger.info("lastHistory=%s", lastHistory)
            result.append(lastHistory.id)

        logger.info("result=%s", result)
        resultLen = len(result)
        logger.info("resultLen=%s", resultLen)
耗时大概:
...

INFO|20180810 23:03:04|views:list:170|===[517] eachHistoryId=c0d1140d-5043-44f4-a1c7-7486f1b0639c
DEBUG|20180810 23:03:04|utils:execute:111|(0.000) SELECT `script_history`.`created_at`, `script_history`.`updated_at`, `script_history`.`id` FROM `script_history` WHERE `script_history`.`id` = 'c0d1140d504344f4a1c77486f1b0639c'; args=('c0d1140d504344f4a1c77486f1b0639c',)
INFO|20180810 23:03:04|views:list:172|history=History object (c0d1140d-5043-44f4-a1c7-7486f1b0639c)
DEBUG|20180810 23:03:04|utils:execute:111|(0.000) SELECT `script_script`.`created_at`, `script_script`.`updated_at`, `script_script`.`id`, `script_script`.`place`, `script_script`.`title`, `script_script`.`topic_id`, `script_script`.`second_level_topic_id`, `script_script`.`age_start`, `script_script`.`age_end`, `script_script`.`version`, `script_script`.`publish_status`, `script_script`.`edit_status`, `script_script`.`review_id`, `script_script`.`history_id`, `script_script`.`author_id` FROM `script_script` WHERE `script_script`.`history_id` = 'c0d1140d504344f4a1c77486f1b0639c' ORDER BY `script_script`.`version` ASC LIMIT 21; args=('c0d1140d504344f4a1c77486f1b0639c',)
INFO|20180810 23:03:04|views:list:174|orderedScriptAllHistory=<QuerySet [<Script: cooking the salad>]>
DEBUG|20180810 23:03:04|utils:execute:111|(0.000) SELECT `script_script`.`created_at`, `script_script`.`updated_at`, `script_script`.`id`, `script_script`.`place`, `script_script`.`title`, `script_script`.`topic_id`, `script_script`.`second_level_topic_id`, `script_script`.`age_start`, `script_script`.`age_end`, `script_script`.`version`, `script_script`.`publish_status`, `script_script`.`edit_status`, `script_script`.`review_id`, `script_script`.`history_id`, `script_script`.`author_id` FROM `script_script` WHERE `script_script`.`history_id` = 'c0d1140d504344f4a1c77486f1b0639c' ORDER BY `script_script`.`version` DESC LIMIT 1; args=('c0d1140d504344f4a1c77486f1b0639c',)
INFO|20180810 23:03:04|views:list:176|lastHistory=cooking the salad
INFO|20180810 23:03:04|views:list:170|===[518] eachHistoryId=8337f985-d113-473f-9bd3-2312c6dc1726
DEBUG|20180810 23:03:04|utils:execute:111|(0.000) SELECT `script_history`.`created_at`, `script_history`.`updated_at`, `script_history`.`id` FROM `script_history` WHERE `script_history`.`id` = '8337f985d113473f9bd32312c6dc1726'; args=('8337f985d113473f9bd32312c6dc1726',)
INFO|20180810 23:03:04|views:list:172|history=History object (8337f985-d113-473f-9bd3-2312c6dc1726)
DEBUG|20180810 23:03:04|utils:execute:111|(0.001) SELECT `script_script`.`created_at`, `script_script`.`updated_at`, `script_script`.`id`, `script_script`.`place`, `script_script`.`title`, `script_script`.`topic_id`, `script_script`.`second_level_topic_id`, `script_script`.`age_start`, `script_script`.`age_end`, `script_script`.`version`, `script_script`.`publish_status`, `script_script`.`edit_status`, `script_script`.`review_id`, `script_script`.`history_id`, `script_script`.`author_id` FROM `script_script` WHERE `script_script`.`history_id` = '8337f985d113473f9bd32312c6dc1726' ORDER BY `script_script`.`version` ASC LIMIT 21; args=('8337f985d113473f9bd32312c6dc1726',)
INFO|20180810 23:03:04|views:list:174|orderedScriptAllHistory=<QuerySet [<Script: xxx>]>
DEBUG|20180810 23:03:04|utils:execute:111|(0.000) SELECT `script_script`.`created_at`, `script_script`.`updated_at`, `script_script`.`id`, `script_script`.`place`, `script_script`.`title`, `script_script`.`topic_id`, `script_script`.`second_level_topic_id`, `script_script`.`age_start`, `script_script`.`age_end`, `script_script`.`version`, `script_script`.`publish_status`, `script_script`.`edit_status`, `script_script`.`review_id`, `script_script`.`history_id`, `script_script`.`author_id` FROM `script_script` WHERE `script_script`.`history_id` = '8337f985d113473f9bd32312c6dc1726' ORDER BY `script_script`.`version` DESC LIMIT 1; args=('8337f985d113473f9bd32312c6dc1726',)
INFO|20180810 23:03:04|views:list:176|lastHistory=xxx
INFO|20180810 23:03:04|views:list:179|result=[UUID('cb54d47d-6e9c-4ec2-8666-eb97df30e654'),
...
INFO|20180810 23:03:04|views:list:181|resultLen=519
(2)序列化耗费了大概5秒 到了23:03:09
serializer = ScriptSerializer(queryset, many=True)
logger.info("after ScriptSerializer serializer=%s", serializer)
serializedData = serializer.data
logger.info("serializedData=%s", serializedData)
输出:
DEBUG|20180810 23:03:09|utils:execute:111|(0.000) SELECT `user_user`.`password`, `user_user`.`last_login`, `user_user`.`is_superuser`, `user_user`.`username`, `user_user`.`first_name`, `user_user`.`last_name`, `user_user`.`email`, `user_user`.`is_staff`, `user_user`.`date_joined`, `user_user`.`id`, `user_user`.`is_active`, `user_user`.`name`, `user_user`.`mobile_phone_number` FROM `user_user` WHERE `user_user`.`id` = '7e8832bcc02d4befa303ed9488fb654a'; args=('7e8832bcc02d4befa303ed9488fb654a',)
INFO|20180810 23:03:09|serializers:to_representation:25|ScriptFuctionGroup to_representation: self=ScriptFuctionGroup(source='author'), value=username=xxx,id=7e8832bc-c02d-4bef-a303-ed9488fb654a,is_superuser=False
INFO|20180810 23:03:09|serializers:to_representation:28|curUser=username=xxx,id=7e8832bc-c02d-4bef-a303-ed9488fb654a,is_superuser=False
DEBUG|20180810 23:03:09|utils:execute:111|(0.000) SELECT `user_functiongroup`.`id`, `user_functiongroup`.`created_at`, `user_functiongroup`.`updated_at`, `user_functiongroup`.`owner_id`, `user_functiongroup`.`name`, `user_functiongroup`.`function`, `user_functiongroup`.`description` FROM `user_functiongroup` INNER JOIN `user_functiongroup_members` ON (`user_functiongroup`.`id` = `user_functiongroup_members`.`functiongroup_id`) WHERE (`user_functiongroup`.`function` = '1' AND `user_functiongroup_members`.`user_id` = '7e8832bcc02d4befa303ed9488fb654a'); args=('1', '7e8832bcc02d4befa303ed9488fb654a')
INFO|20180810 23:03:09|serializers:to_representation:31|joinedScriptGroup=script_function_group
DEBUG|20180810 23:03:09|utils:execute:111|(0.000) SELECT COUNT(*) AS `__count` FROM `script_dialog` WHERE `script_dialog`.`script_id` = '9215ab9492f34e3b8f02677743e05e3b'; args=('9215ab9492f34e3b8f02677743e05e3b',)
INFO|20180810 23:03:09|views:list:191|serializedData=[OrderedDict([('id',
所以结论是:
代码中业务逻辑有点复杂,加上原先设计的不够好,导致此处 检索4秒+序列化5秒,加起来需要9秒左右才返回到前端 -》 误以为前端有问题呢。
所以接下来,就是去优化业务逻辑和设计,以便于减少代码时间。
其中可以做的是,现在的序列化是针对所有的数据的,
应该改为:只针对于需要返回的那些数据即可。
以及,再去想办法通过优化提升速度:
根据已有业务逻辑,在不大动干戈改动数据库的情况下(因为直接给Script增加,去优化减少查询数据库,最后代码优化改为:
from django.conf import settings
from rest_framework.response import Response
from django.core.paginator import Paginator


class ScriptViewSet(mixins.ListModelMixin,
                    mixins.CreateModelMixin,
                    mixins.RetrieveModelMixin,
                    PutOnlyUpdateModelMixin,
                    mixins.DestroyModelMixin,
                    viewsets.GenericViewSet):
    queryset = Script.objects.all()
    serializer_class = ScriptSerializer
    permission_classes = (IsAuthenticated, IsUserScriptFunctionGroup)
...
    def list(self, request, *args, **kwargs):
    ...
        # filterByUserScriptList = Script.objects.filter(userFilter)
        # Note: here order by created time to let new created show first
        # -> make it easy for later will got latest version script via filter by history id
        filterByUserScriptList = Script.objects.filter(userFilter).order_by('-created_at')
        logger.info("filterByUserScriptList=%s", filterByUserScriptList)
        filterByUserScriptListLen = len(filterByUserScriptList)
        logger.info("filterByUserScriptListLen=%s", filterByUserScriptListLen)

        filter_condition = self.generateQueryFilterCondiction(request)
        logger.info("filter_condition=%s", filter_condition)
        resultScriptIdList = []

        latestVersionDict = {} # store history_id : latest_version_script
        # uniqueHistoryIdList = []
        for curScriptIdx, singleScript in enumerate(filterByUserScriptList):
            logger.info("---[%d] singleScript=%s", curScriptIdx, singleScript)
            scriptHistoryId = singleScript.history_id
            logger.info("scriptHistoryId=%s", scriptHistoryId)
            # if scriptHistoryId not in uniqueHistoryIdList:
            if scriptHistoryId not in latestVersionDict.keys():
                # uniqueHistoryIdList.append(singleScript.history_id)
                # latestVersionScriptList.append(singleScript)
                latestVersionDict[scriptHistoryId] = singleScript
            else:
                # logger.debug("filter out [%d] script: %s", curScriptIdx, singleScript)
                logger.debug("Check is latest version or not for: [%d] singleScript=%s", curScriptIdx, singleScript)
                prevStoredScript = latestVersionDict[scriptHistoryId]
                logger.debug("prevStoredScript=%s", prevStoredScript)
                prevStoredScriptVersion = prevStoredScript.version
                curScriptVersion = singleScript.version
                logger.debug("prevStoredScriptVersion=%d, curScriptVersion=%d", prevStoredScriptVersion, curScriptVersion)
                if (curScriptVersion > prevStoredScriptVersion):
                    latestVersionDict[scriptHistoryId] = singleScript
                else:
                    logger.debug("omit older version script: %s", singleScript)

        # generate result script id list
        for eachHistoryId in latestVersionDict.keys():
            logger.debug("eachHistoryId=%s", eachHistoryId)
            eachScript = latestVersionDict[eachHistoryId]
            logger.debug("eachScript=%s", eachScript)
            resultScriptIdList.append(eachScript.id)

        # logger.info("uniqueHistoryIdList=%s", uniqueHistoryIdList)
        # uniqueHistoryIdListLen = len(uniqueHistoryIdList)
        # logger.info("uniqueHistoryIdListLen=%s", uniqueHistoryIdListLen)

        # for curHisotryIdIdx, eachHistoryId in enumerate(uniqueHistoryIdList):
        #     logger.info("===[%d] eachHistoryId=%s", curHisotryIdIdx, eachHistoryId)
        #     history = History.objects.get(pk=eachHistoryId)
        #     logger.info("history=%s", history)
        #     orderedScriptAllHistory = history.script_history.all().order_by('version')
        #     logger.info("orderedScriptAllHistory=%s", orderedScriptAllHistory)
        #     lastHistory = orderedScriptAllHistory.last()
        #     logger.info("lastHistory=%s", lastHistory)
        #     resultScriptIdList.append(lastHistory.id)

        logger.info("resultScriptIdList=%s", resultScriptIdList)
        resultScriptIdListLen = len(resultScriptIdList)
        logger.info("resultScriptIdListLen=%s", resultScriptIdListLen)

        allScriptList = Script.objects.filter(pk__in=resultScriptIdList).filter(filter_condition).order_by('-created_at')
        logger.info("allScriptList=%s", allScriptList)

        # paginatedQueryset = self.paginate_queryset(allScriptList)
        # logger.info("paginatedQueryset=%s", paginatedQueryset)

        # serializer = ScriptSerializer(allScriptList, many=True)
        # logger.info("after ScriptSerializer serializer=%s", serializer)
        # serializedData = serializer.data
        # logger.info("serializedData=%s", serializedData)

        # respDict = None
        # if paginatedQueryset is not None:
        #     respDict = self.get_paginated_response(serializedData)
        #     # respDict = self.get_paginated_response(serializedData, page)
        # else:
        #     respDict = Response(serializedData)

        # logger.info("respDict=%s", respDict)
        # return respDict

        # curPaginator = Paginator(serializedData, page_size)
        curPaginator = Paginator(allScriptList, page_size)
        logger.info("curPaginator=%s", curPaginator)
        totalCount = curPaginator.count
        logger.info("totalCount=%s", totalCount)
        maxPageCount = curPaginator.num_pages
        logger.info("maxPageCount=%s", maxPageCount)

        curPageNum = page
        logger.info("curPageNum=%s", curPageNum)
        curPage = curPaginator.page(curPageNum)
        logger.info("curPage=%s", curPage)
        logger.info("type(curPage)=%s", type(curPage))

        curPageItemList = curPage.object_list
        logger.info("curPageItemList=%s", curPageItemList)

        curPageSerializer = ScriptSerializer(curPageItemList, many=True)
        logger.info("curPageSerializer=%s", curPageSerializer)
        curPageSerializedData = curPageSerializer.data
        logger.info("curPageSerializedData=%s", curPageSerializedData)

        # currentPageCount = len(curPageItemList)
        currentPageCount = len(curPageSerializedData)
        logger.info("currentPageCount=%s", currentPageCount)

        # nextPageUrl = self.get_next_link()
        # previousPageUrl = self.get_previous_link()
        nextPageUrl = None
        previousPageUrl = None

        respDict = {
            "totalCount": totalCount,
            "maxPageCount": maxPageCount,
            "pageSize": page_size,
            "currentPageNumber": curPageNum,
            "currentPageCount": currentPageCount,
            "next": nextPageUrl,
            "previous": previousPageUrl,
            # "results": curPageItemList,
            "results": curPageSerializedData,
        }

        return Response(respDict, status=status.HTTP_200_OK)
从而把之前的耗时9秒左右(4秒的查询script的hisotry + 5秒的所有页面数据的序列化serialize)的操作,优化不到1秒:
【后记】
后来看到:
聊聊python的轻量级orm peewee及对比SQLAlchemy | 峰云就她了
http://xiaorui.cc/2015/10/09/聊聊python的轻量级orm-peewee及对比sqlalchemy/
-》
话说Django orm性能为什么比原生的mysqldb慢 | 峰云就她了
http://xiaorui.cc/2015/09/24/话说django-orm模型为什么比原生的mysqldb慢/
其中遇到Django的ORM性能慢的现象:
感觉和此处很类似:
上面的Script的History的之前代码,也是通过Django的ORM去跨表查询的:
history = History.objects.get(pk=eachHistoryId)
orderedScriptAllHistory = history.script_history.all().order_by('version')
lastHistory = orderedScriptAllHistory.last()
500多条Script,通过script_history的外键去查询所有符合条件的内容,再去找到last最新的一条
也是查询很慢:大概要花4秒左右
而自己当时的优化是通过逻辑上避免了这个额外的查询。
看来如果以后会有机会,借鉴其所说的:
“直接走原生的mysql sql语句,在python下你的选择 mysqldb,也可以用django的connection。推荐用connection,因为大家的db地址配置都是放在settings.config里面的。”
即:也是可以通过直接转换为内部的SQL查询语句,直接查询,提升性能的。

转载请注明:在路上 » 【已解决】Antd Pro中前端列表页面loading加载很慢

发表我的评论
取消评论

表情

Hi,您需要填写昵称和邮箱!

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址
87 queries in 0.113 seconds, using 20.79MB memory