最新消息:20210917 已从crifan.com换到crifan.org

【已解决】批量处理gitbook的脚本时忽略掉已处理的book

gitbook crifan 951浏览 0评论
折腾:
【记录】给crifan的gitbook的books去批量make deploy部署
期间,之前弄的:
【已解决】批量编译和发布gitbook电子书 – 在路上
脚本可以批量的去
make deploy
了。但是遇到问题:
偶尔会报错:
然后如果重新
make deploy
又会重头开始,很麻烦。
所以希望:支持指定起始book,这样就可以跳过之前已处理过的book了。
去看看,希望可以指定起始的要处理的book
这样就可以跳过该book之前的book了。
不过也去看看此处的问题:
【已解决】git pull报错:Auto packing the repository fatal failed to run repack
此处脚本中:
SUB_BOOKS=$(shell ls ../books)
BOOKS_SRC_ROOT=$(shell cd ../books && pwd)


# Batch make for all gitbooks
help debug_dir init sync_content clean_all website pdf epub mobi all upload commit deploy:
    @echo "Current path="`pwd`;
    @echo "LS_OUTPUT="$(SUB_BOOKS);
    @echo "BOOKS_SRC_ROOT="$(BOOKS_SRC_ROOT);
    @for each_item in $(SUB_BOOKS); \
    do \
        if [ -d $(BOOKS_SRC_ROOT)/$$each_item ]; then \
            cd $(BOOKS_SRC_ROOT)/$$each_item; \
            echo `pwd`; \
            if [ -f Makefile ]; then \
                make $@ || exit "$$?"; \
            fi; \
            cd ..; \
        fi; \
    done;
核心就是for循环
如何在for循环中指定起始的,不好指定
突然想到,或许可以通过SUB_BOOKS,去掉某个book之前的?
makefile str list
linux – How to loop on array of string – Stack Overflow
Text Functions (GNU make)
没有合适的函数
Functions (GNU make)
Shell Function (GNU make)
shell split list
想到了:
或许可以在for循环中,加bool变量,在start之前 都忽略,等于 设置为true,后续发现true,再处理
bash bool value
bash – How can I declare and use Boolean variables in a shell script? – Stack Overflow
makefile bool value
gnu make – Variable in makefile conditional? – Stack Overflow
【已解决】Makefile中如何定义和判断布尔类型变量 – 在路上
makefile for loop
How to write loop in a Makefile? – Stack Overflow
bash – simple loop over files in some directory makefile – Stack Overflow
shell – How to use a for loop in make recipe – Stack Overflow
Makefile:Loop循环的使用 – 简书
bash if else fi
Bash if..else Statement | Linuxize
if [[ $VAR -gt 10 ]]
then
  echo "The variable is greater than 10."
else
  echo "The variable is equal or less than 10."
fi
4 Bash If Statement Examples ( If then fi, If then else fi, If elif else fi, Nested if )
bash inside if set value
if statement – BASH: Basic if then and variable assignment – Stack Overflow
Unix Bash – Assign if/else to Variable – Stack Overflow
bash set value
Variable Assignment
BASH Programming – Introduction HOW-TO: Variables
How to Work with Variables in Bash
折腾了半天:
START_BOOK = automobile_sales_summary


SUB_BOOKS=$(shell ls ../books)
BOOKS_SRC_ROOT=$(shell cd ../books && pwd)


IS_PROCESS = false


ifeq ($(START_BOOK), )
IS_PROCESS = true
$(info START_BOOK is empty, now IS_PROCESS=$(IS_PROCESS))
else
IS_PROCESS = false
$(info START_BOOK=$(START_BOOK), not empty, now IS_PROCESS=$(IS_PROCESS))
endif


# Batch make for all gitbooks
help debug_dir init sync_content clean_all website pdf epub mobi all upload commit deploy:
    @echo "Current path="`pwd`;
    @# echo "LS_OUTPUT="$(SUB_BOOKS);
    @echo "BOOKS_SRC_ROOT="$(BOOKS_SRC_ROOT);
    @for each_item in $(SUB_BOOKS); \
    do \
        if [ $IS_PROCESS = true ] ; then \
            echo IS_PROCESS is true, set to false; \
            $IS_PROCESS=false; \
        else \
            echo IS_PROCESS is false, set to true; \
            $IS_PROCESS=true; \
        fi; \
        if [ -d $(BOOKS_SRC_ROOT)/$$each_item ]; then \
            cd $(BOOKS_SRC_ROOT)/$$each_item; \
            # echo `pwd`; \
            if [ -f Makefile ]; then \
                echo $$each_item; \
                # make $@ || exit "$$?"; \
            fi; \
            cd ..; \
        fi; \
    done;
还是没有实现要的效果。
放弃。
另外,也想到了,用忽略omit的方式,但是觉得太麻烦。还要挨个去拷贝book的name,麻烦。算了。
后来遇到:运行了一晚上,结果还是由于错误停在某个book
导致后续重新全部运行,还是太慢,无法接受,所以还是去加上吧
即使是omit list,也比没有强
去加上 忽略列表,在列表中的,不处理
makfile findstring
Linux之Makefile(findstring)_zhoudengqing的专栏-CSDN博客
使用函数 — 跟我一起写Makefile 1.0 文档
用:
BATCH_PROCESS_IGNORE_FILE=batch_process_ignore_books.txt
$(info BATCH_PROCESS_IGNORE_FILE=$(BATCH_PROCESS_IGNORE_FILE))
$(info ----------------------------------------)


BATCH_PROCESS_IGNORE_BOOKS := $(shell cat $(BATCH_PROCESS_IGNORE_FILE))
$(info BATCH_PROCESS_IGNORE_BOOKS=$(BATCH_PROCESS_IGNORE_BOOKS))
$(info ----------------------------------------)


SUB_BOOKS=$(shell ls ../../books)
$(info SUB_BOOKS=$(SUB_BOOKS))
$(info ----------------------------------------)


BOOKS_SRC_ROOT=$(shell cd ../../books && pwd)
$(info BOOKS_SRC_ROOT=$(BOOKS_SRC_ROOT))
$(info ----------------------------------------)


TO_PROCESS_BOOKS := $(filter-out $(BATCH_PROCESS_IGNORE_BOOKS), $(SUB_BOOKS))
$(info TO_PROCESS_BOOKS=$(TO_PROCESS_BOOKS))
$(info ----------------------------------------)


help:
    echo help;
已经可以基本上输出我们要的了:
●  make help
BATCH_PROCESS_IGNORE_FILE=batch_process_ignore_books.txt
----------------------------------------
BATCH_PROCESS_IGNORE_BOOKS=5g_message_rcs_tech_summary 5g_tech_summary all_age_sports_badminton android_app_security_crack android_automation_uiautomator2 api_tool_postman app_capture_package_tool_charles apple_develop_summary automobile_sales_summary best_editor_vscode china_suitable_living_suzhou common_logic_hardware_embedded computer_tech_summary crack_assistant_xposed_framework crawl_your_data_spider_technology desktop_app_framework_electron doc_format_markdown ebook_system_gitbook editor_ide_summary gitbook_demo good_automation_tool_makefile html_parse_tool_beautifulsoup http_restful_api http_summary ic_chip_industry_chain_summary improve_work_efficiency industrial_automation_plc industrial_control_security_overview information_security_overview ios_automation_facebook_wda learn_tech_method_experience mobile_app_summary mobile_automation_overview mobile_network_evolution_history most_intelligent_python_ide_pycharm multimedia_core_system_ims popular_document_db_mongodb program_code_style program_common_logic
----------------------------------------
SUB_BOOKS=5g_message_rcs_tech_summary 5g_tech_summary all_age_sports_badminton android_app_security_crack android_automation_uiautomator2 api_tool_postman app_capture_package_tool_charles apple_develop_summary automobile_sales_summary best_editor_vscode china_suitable_living_suzhou common_logic_hardware_embedded computer_tech_summary convertToNewCommonTemplate.sh crack_assistant_xposed_framework crawl_your_data_spider_technology desktop_app_framework_electron doc_format_markdown ebook_system_gitbook editor_ide_summary gitbook_demo good_automation_tool_makefile html_parse_tool_beautifulsoup http_restful_api http_summary ic_chip_industry_chain_summary improve_work_efficiency industrial_automation_plc industrial_control_security_overview information_security_overview ios_automation_facebook_wda learn_tech_method_experience make_life_better_python mobile_app_summary mobile_automation_overview mobile_network_evolution_history most_intelligent_python_ide_pycharm multimedia_core_system_ims popular_document_db_mongodb program_code_style program_common_logic python_common_code_snippet python_experience_csv_excel python_html_parse_pyquery python_newbie_mistakes_questions python_regex_re_intro python_spider_pyspider python_spider_scrapy rcs_tech_dev_summary regex_usage_examples resident_life_experience_summary scientific_network_summary selenium_summary smart_speaker_disassemble_summary super_search_regex use_python_write_spider work_job_summary xpath_summary youdao_note_summary
----------------------------------------
BOOKS_SRC_ROOT=/Users/crifan/dev/dev_root/gitbook/GitbookTemplate/gitbook_template/books
----------------------------------------
TO_PROCESS_BOOKS=convertToNewCommonTemplate.sh make_life_better_python python_common_code_snippet python_experience_csv_excel python_html_parse_pyquery python_newbie_mistakes_questions python_regex_re_intro python_spider_pyspider python_spider_scrapy rcs_tech_dev_summary regex_usage_examples resident_life_experience_summary scientific_network_summary selenium_summary smart_speaker_disassemble_summary super_search_regex use_python_write_spider work_job_summary xpath_summary youdao_note_summary
----------------------------------------
echo help;
help
但是还要去除特殊的 非book的:convertToNewCommonTemplate.sh
本来要加上special,去除掉的
但是不够智能
希望去加上更智能的:符合一定规则的
就像正则的\w+的,不符合的,都剔除
去找找 makefile 或者 shell中是否有正则的写法
dir
https://seisman.github.io/how-to-write-makefile/functions.html#dir
notdir
https://seisman.github.io/how-to-write-makefile/functions.html#notdir
发现可以用:判断是否是dir的方式去实现过滤
suffix
https://seisman.github.io/how-to-write-makefile/functions.html#suffix
或者判断有suffix,则过滤掉
还是判断是否是dir更合适
看错了,dir是返回path的dir部分。。。
不过可以去for循环中匹配出,非-d的部分
不过突然发现之前过来结果,好像不对
好像多出个,在program_common_logic之前的:make_life_better_python
不过再去看看,发现是:
的确之前漏了:make_life_better_python
generated中是没有make_life_better_python
算了,暂时就这样吧,通过-d判断:
if [ -d $(BOOKS_SRC_ROOT)/$$each_item ];
即:
help:
    @# echo help;
    @for each_item in $(TO_PROCESS_BOOKS); \
    do \
        if [ -d $(BOOKS_SRC_ROOT)/$$each_item ]; then \
            echo $$each_item; \
        fi; \
    done;
可以忽略掉。
makefile check is folder
bash – Test whether a directory exists inside a makefile – Stack Overflow
REAL_ALL_BOOKS = $(foreach singleBook,$(SUB_BOOKS),$(if [-d $(BOOKS_SRC_ROOT)/$(SUB_BOOKS)],$(SUB_BOOKS),)
$(info REAL_ALL_BOOKS=$(REAL_ALL_BOOKS))
$(info ----------------------------------------)
看看能否实现过滤掉非folder的book
Makefile:27: *** unterminated call to function 'foreach': missing ')'.  Stop.
报错。
REAL_ALL_BOOKS = $(foreach singleBook,$(SUB_BOOKS),$(if [-d $(BOOKS_SRC_ROOT)/$(SUB_BOOKS)],$(SUB_BOOKS))
结果:错误依旧
继续试试:
REAL_ALL_BOOKS = $(foreach singleBook,$(SUB_BOOKS),$(if [-d $(BOOKS_SRC_ROOT)/$(SUB_BOOKS)],$(SUB_BOOKS)))
写错成SUB_BOOKS,改为:
REAL_ALL_BOOKS = $(foreach singleBook,$(SUB_BOOKS),$(if [-d $(BOOKS_SRC_ROOT)/$(singleBook)],$(singleBook)))
可以运行,但没过滤掉convertToNewCommonTemplate.sh
REAL_ALL_BOOKS = $(foreach singleBook,$(SUB_BOOKS),$(if [-d $(BOOKS_SRC_ROOT)/$(singleBook)],$(singleBook),))
结果:问题依旧。
test if directory exists in a makefile · GitHub
Makefile: Test directories for existence and take first one found – Stack Overflow
试了试:
 crifan@licrifandeMacBook-Pro  ~/dev/dev_root/gitbook/GitbookTemplate/gitbook_template/common/batchProcess   master ●  ls -d ../../books
../../books
 crifan@licrifandeMacBook-Pro  ~/dev/dev_root/gitbook/GitbookTemplate/gitbook_template/common/batchProcess   master ●  ls -d ../../books -d
ls: -d: No such file or directory
../../books
 ✘ crifan@licrifandeMacBook-Pro  ~/dev/dev_root/gitbook/GitbookTemplate/gitbook_template/common/batchProcess   master ●  ls --help           
ls: illegal option -- -
usage: ls [-@ABCFGHLOPRSTUWabcdefghiklmnopqrstuwx1%] [file ...]
ls的-d,不可以?
ls only folder
directory – Listing only directories using ls in Bash? – Stack Overflow
directory – Listing only directories using ls in Bash? – Stack Overflow
●  ls -l ../../books | grep "^d"
drwxr-xr-x  11 crifan  staff  352  1 16 23:43 5g_message_rcs_tech_summary
drwxr-xr-x  11 crifan  staff  352  1 16 23:45 5g_tech_summary
...
但是前面多余了,不方便处理只保留book的name
  find ../../books -maxdepth 1 -type d
../../books
../../books/computer_tech_summary
../../books/mobile_app_summary
../../books/editor_ide_summary
也不是
shell – Listing only directories in UNIX – Stack Overflow
shell – list only folders (ls -d) relative to parent in bash – Stack Overflow
Mac ls only folder
utilities – Is there any option with ‘ls’ command that I see only the directories? – Unix & Linux Stack Exchange
 ls -ld ../../books           
drwxr-xr-x  62 crifan  staff  1984  1 16 11:58 ../../books
ls parameter
ls Command With All Its Parameter In Linux | Geekstrick
ls man page
       -d, --directory
              list directories themselves, not their contents


       -D, --dired
              generate output designed for Emacs' dired mode
ls command in Linux/Unix | list files/directories
ls -d
list directories - with ' */'
试试:
ls -d */
结果:不行。
makefile ifeq
makefile中的条件判断ifeq、ifneq、ifdef_nyist327的专栏-CSDN博客
Conditional Example (GNU make)
makefile inside for check folder
how to prevent “directory already exists error” in a makefile when using mkdir – Stack Overflow
REAL_ALL_BOOKS = $(foreach singleBook,$(SUB_BOOKS),$(if [test -d $(BOOKS_SRC_ROOT)/$(singleBook)],,$(singleBook)))
How can I check if directory exists in a makefile
【总结】
最后代码是:
# Function: Batch make for all gitbooks
# Author: Crifan Li
# Update: 20210117
# Latest: https://github.com/crifan/gitbook_template/blob/master/common/Makefile
# 
# [Note]
# 1. this makefile should be located common folder:
# /Users/crifan/dev/dev_root/gitbook/GitbookTemplate/gitbook_template/common/Makefile
# /Users/crifan/dev/dev_root/gitbook/gitbook_src_root/common


BATCH_PROCESS_IGNORE_FILE=batch_process_ignore_books.txt
$(info BATCH_PROCESS_IGNORE_FILE=$(BATCH_PROCESS_IGNORE_FILE))
$(info ----------------------------------------)


BATCH_PROCESS_IGNORE_BOOKS := $(shell cat $(BATCH_PROCESS_IGNORE_FILE))
$(info BATCH_PROCESS_IGNORE_BOOKS=$(BATCH_PROCESS_IGNORE_BOOKS))
$(info ----------------------------------------)


SUB_BOOKS=$(shell ls ../../books)
$(info SUB_BOOKS=$(SUB_BOOKS))
$(info ----------------------------------------)


BOOKS_SRC_ROOT=$(shell cd ../../books && pwd)
$(info BOOKS_SRC_ROOT=$(BOOKS_SRC_ROOT))
$(info ----------------------------------------)


# # REAL_ALL_BOOKS = $(foreach singleBook,$(SUB_BOOKS),$(if [-d $(BOOKS_SRC_ROOT)/$(singleBook)],$(singleBook),))
# # REAL_ALL_BOOKS = $(foreach singleBook,$(SUB_BOOKS),$(if [-f $(BOOKS_SRC_ROOT)/$(singleBook)],,$(singleBook)))
# REAL_ALL_BOOKS = $(foreach singleBook,$(SUB_BOOKS),$(if [test -d $(BOOKS_SRC_ROOT)/$(singleBook)],$(singleBook),))
# $(info REAL_ALL_BOOKS=$(REAL_ALL_BOOKS))
# $(info ----------------------------------------)


TO_PROCESS_BOOKS := $(filter-out $(BATCH_PROCESS_IGNORE_BOOKS), $(SUB_BOOKS))
$(info TO_PROCESS_BOOKS=$(TO_PROCESS_BOOKS))
$(info ----------------------------------------)


# help:
#     @# echo help;
#     @for each_item in $(TO_PROCESS_BOOKS); \
#     do \
#         if [ -d $(BOOKS_SRC_ROOT)/$$each_item ]; then \
#             echo $$each_item; \
#         fi; \
#     done;


# Batch make for all gitbooks
help debug_dir init sync_content pull clean_all website pdf epub mobi all upload commit deploy:
    @echo "Current path="`pwd`;
    @echo "LS_OUTPUT="$(TO_PROCESS_BOOKS);
    @echo "BOOKS_SRC_ROOT="$(BOOKS_SRC_ROOT);
    @for each_item in $(TO_PROCESS_BOOKS); \
    do \
        if [ -d $(BOOKS_SRC_ROOT)/$$each_item ]; then \
            cd $(BOOKS_SRC_ROOT)/$$each_item; \
            # echo `pwd`; \
            if [ -f Makefile ]; then \
                # echo $$each_item; \
                make $@ || exit "$$?"; \
            fi; \
            cd ..; \
        fi; \
    done;
可以实现:
通过设置:
/Users/crifan/dev/dev_root/gitbook/GitbookTemplate/gitbook_template/common/batchProcess/batch_process_ignore_books.txt
5g_message_rcs_tech_summary
5g_tech_summary
all_age_sports_badminton
android_app_security_crack
android_automation_uiautomator2
api_tool_postman
app_capture_package_tool_charles
apple_develop_summary
automobile_sales_summary
best_editor_vscode
china_suitable_living_suzhou
common_logic_hardware_embedded
computer_tech_summary
crack_assistant_xposed_framework
crawl_your_data_spider_technology
desktop_app_framework_electron
doc_format_markdown
ebook_system_gitbook
editor_ide_summary
gitbook_demo
good_automation_tool_makefile
html_parse_tool_beautifulsoup
http_restful_api
http_summary
ic_chip_industry_chain_summary
improve_work_efficiency
industrial_automation_plc
industrial_control_security_overview
information_security_overview
ios_automation_facebook_wda
learn_tech_method_experience
mobile_app_summary
mobile_automation_overview
mobile_network_evolution_history
most_intelligent_python_ide_pycharm
multimedia_core_system_ims
popular_document_db_mongodb
program_code_style
program_common_logic
然后批量处理是,忽略掉这些book
另外其中的:
if [ -d $(BOOKS_SRC_ROOT)/$$each_item ]; then \
通过仅仅处理(正常book都是)目录
实现过滤掉:
/Users/crifan/dev/dev_root/gitbook/GitbookTemplate/gitbook_template/books/
下面的 非book内容
比如此处的:
convertToNewCommonTemplate.sh
防止误处理非book
【后记20220225】
相关资料:
Linux Commands Cheat Sheet – Quickly Reference Common Commands! (pcwdld.com)

转载请注明:在路上 » 【已解决】批量处理gitbook的脚本时忽略掉已处理的book

发表我的评论
取消评论

表情

Hi,您需要填写昵称和邮箱!

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址
92 queries in 0.192 seconds, using 23.43MB memory