源代码在主页https://www.xunsearch.com/scws/download.php 和 https://github.com/hightman/scws/ 下载都行1.编译我从github.com下载的源文件根目录中没有configure文件需要用autoreconf工具生成宿主机没有此工具转到gcc docker容器执行。tar xf scws-1.2.3.tar.gz cd scws-1.2.3 ./configure bash: ./configure: 没有那个文件或目录 ls acprep API.md AUTHORS ChangeLog cli configure.ac COPYING dev etc libscws Makefile.am NEWS phpext README.md web win32 autoconf ./configure.ac Could not find command-not-found database. Run sudo apt update to populate it. autoconf未找到命令 sudo docker start gcc142 输入密码 gcc142 sudo docker exec -it gcc142 bash cd /par/scws* autoreconf -ivf autoreconf: export WARNINGS autoreconf: Entering directory . autoreconf: configure.ac: not using Gettext autoreconf: running: aclocal --force autoreconf: configure.ac: tracing autoreconf: running: libtoolize --copy --force libtoolize: putting auxiliary files in .. libtoolize: copying file ./ltmain.sh libtoolize: Consider adding AC_CONFIG_MACRO_DIRS([m4]) to configure.ac, libtoolize: and rerunning libtoolize and aclocal. libtoolize: Consider adding -I m4 to ACLOCAL_AMFLAGS in Makefile.am. autoreconf: configure.ac: not using Intltool autoreconf: configure.ac: not using Gtkdoc autoreconf: running: aclocal --force autoreconf: running: /usr/bin/autoconf --force ... ./lib/autoconf/general.m4:2847: AC_TRY_COMPILE is expanded from... configure.ac:80: the top level autoreconf: running: /usr/bin/autoheader --force autoreconf: running: automake --add-missing --copy --force-missing configure.ac:23: installing ./compile configure.ac:29: installing ./config.guess configure.ac:29: installing ./config.sub configure.ac:9: installing ./install-sh configure.ac:9: installing ./missing Makefile.am: installing ./INSTALL cli/Makefile.am:8: warning: INCLUDES is the old name for AM_CPPFLAGS (or *_CPPFLAGS) cli/Makefile.am: installing ./depcomp autoreconf: Leaving directory .官方网站下载的scws-1.2.3.tar.bz2文件中有configure文件可以略过上面这步。然后就可以按文档写的步骤执行了。./configure checking for a BSD-compatible install... /usr/bin/install -c checking whether build environment is sane... yes checking for a race-free mkdir -p... /usr/bin/mkdir -p checking for gawk... no checking for mawk... mawk checking whether make sets $(MAKE)... yes checking whether make supports nested variables... yes checking for gcc... gcc checking whether the C compiler works... yes checking for C compiler default output file name... a.out checking for suffix of executables... ... checking that generated files are newer than configure... done configure: creating ./config.status config.status: creating Makefile config.status: creating cli/Makefile config.status: creating etc/Makefile config.status: creating libscws/Makefile config.status: creating libscws/version.h config.status: creating config.h config.status: executing depfiles commands config.status: executing libtool commands make make all-recursive make[1]: Entering directory /par/scws-1.2.3 Making all in . make[2]: Entering directory /par/scws-1.2.3 make[2]: Leaving directory /par/scws-1.2.3 Making all in libscws make[2]: Entering directory /par/scws-1.2.3/libscws /bin/bash ../libtool --preserve-dup-deps --tagCC --modecompile gcc -DHAVE_CONFIG_H -I. -I.. -g -O2 -MT charset.lo -MD -MP -MF .deps/charset.Tpo -c -o charset.lo charset.c libtool: compile: gcc -DHAVE_CONFIG_H -I. -I.. -g -O2 -MT charset.lo -MD -MP -MF .deps/charset.Tpo -c charset.c -fPIC -DPIC -o .libs/charset.o ... libtool: link: gcc -shared -fPIC -DPIC .libs/charset.o .libs/crc32.o .libs/pool.o .libs/scws.o .libs/xdict.o .libs/darray.o .libs/rule.o .libs/lock.o .libs/xdb.o .libs/xtree.o -lm -g -O2 -Wl,-soname -Wl,libscws.so.1 -o .libs/libscws.so.1.1.0 libtool: link: (cd .libs rm -f libscws.so.1 ln -s libscws.so.1.1.0 libscws.so.1) libtool: link: (cd .libs rm -f libscws.so ln -s libscws.so.1.1.0 libscws.so) libtool: link: ( cd .libs rm -f libscws.la ln -s ../libscws.la libscws.la ) make[2]: Leaving directory /par/scws-1.2.3/libscws Making all in cli make[2]: Entering directory /par/scws-1.2.3/cli gcc -DHAVE_CONFIG_H -I. -I.. -I.. -I../libscws -g -O2 -MT scws_cmd.o -MD -MP -MF .deps/scws_cmd.Tpo -c -o scws_cmd.o scws_cmd.c mv -f .deps/scws_cmd.Tpo .deps/scws_cmd.Po ... make[2]: Leaving directory /par/scws-1.2.3/cli Making all in etc make[2]: Entering directory /par/scws-1.2.3/etc make[2]: Nothing to be done for all. make[2]: Leaving directory /par/scws-1.2.3/etc make[1]: Leaving directory /par/scws-1.2.3 make install Making install in . make[1]: Entering directory /par/scws-1.2.3 make[2]: Entering directory /par/scws-1.2.3 ... libtool: finish: PATH/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/sbin ldconfig -n /usr/local/lib ---------------------------------------------------------------------- Libraries have been installed in: /usr/local/lib If you ever happen to want to link against installed libraries in a given directory, LIBDIR, you must either use libtool, and specify the full pathname of the library, or use the -LLIBDIR flag during linking and do at least one of the following: - add LIBDIR to the LD_LIBRARY_PATH environment variable during execution - add LIBDIR to the LD_RUN_PATH environment variable during linking - use the -Wl,-rpath -Wl,LIBDIR linker flag - have your system administrator add LIBDIR to /etc/ld.so.conf See any operating system documentation about shared libraries for more information, such as the ld(1) and ld.so(8) manual pages. ---------------------------------------------------------------------- /usr/bin/mkdir -p /usr/local/include/scws /usr/bin/install -c -m 644 charset.h crc32.h pool.h scws.h xdict.h darray.h rule.h xdb.h xtree.h version.h /usr/local/include/scws make[2]: Leaving directory /par/scws-1.2.3/libscws make[1]: Leaving directory /par/scws-1.2.3/libscws Making install in cli make[1]: Entering directory /par/scws-1.2.3/cli make[2]: Entering directory /par/scws-1.2.3/cli /usr/bin/mkdir -p /usr/local/bin /bin/bash ../libtool --preserve-dup-deps --modeinstall /usr/bin/install -c scws scws-gen-dict /usr/local/bin ... make[1]: Leaving directory /par/scws-1.2.3/etcmake install把动态库文件放到了/usr/local/lib需要把它加入LD_LIBRARY_PATH环境变量。然后就能执行CLI了, 命令行参数如下。export LD_LIBRARY_PATH$LD_LIBRARY_PATH:/usr/local/lib scws -h scws (scws-cli/1.2.3) Simple Chinese Word Segmentation - Command line usage. Copyright (C)2007 by hightman. Usage: scws [options] [input] [output] -i file|string input string or filepath (default: try to read from stdin everyline) -o file output filepath (default to stdout) -c charset set the charset (default: gbk) charset must been same with dictionary ruleset -r file set the ruleset file (default: none) -d file set the dictionary file[s] (default: none) if there are multi files, split filepath use : if the file suffix is .txt, it will be treated as plain text dict. -M 1~15 use multi child words mode(中国人-中国人中国人) 1|2|4|8: short|duality|zmain|zall -I ignore the all mark symbol such as ,: -A show the word attribute -E import the xdb dict into xtree(memory) -N dont show time usage and warnings -D debug segment, see the segment detail -U use duality algorithm for single chinese -t NUM fetch the top words instead of segment -a [~]attr1,attr2,... prefix by ~ means exclude them. For topwords, exclude or include some word attrs -v Show the version. -h Show this page for help. Report bugs to hightman2yahoo.com.cn注意中文文件分词需要指定中文字典否则把每个汉字都作为1个词了。前10个高频词都是英文数字scws -c utf8 -i 155.txt .... 实 协 同 监 督 机 制 。 规 划 实 施 情 况 及 时 公 开 接 受 社 会 各 界 监 督 。 --[scws(scws-cli/1.2.3)]---------- | TextLen: 181431 | | Prepare: 0.0000 (sec) | | Segment: 0.0309 (sec) | -------------------------------- scws -c utf8 -i 155.txt -t 10 -M 8 No. WordString Attr Weight(times) ------------------------------------------------- 01. 2035 en 10.40(3) 02. PM2 en 8.24(3) 03. 2026 en 6.93(2) 04. 2030 en 6.93(2) 05. 10% en 5.49(2) 06. GDP en 5.49(2) 07. 10.5% en 4.02(1) 08. 25.8% en 4.02(1) 09. REITs en 4.02(1) 10. 1.45 en 3.47(1) --[scws(scws-cli/1.2.3)]---------- | TextLen: 181431 | | Prepare: 0.0000 (sec) | | Segment: 0.0050 (sec) | --------------------------------2.安装字典github源代码包中有个scws-1.2.3.tar.gz\scws-1.2.3\dev\php4\pscws23\dict\dict.xdb.gz, 但它是损坏的。gzip -d dict.xdb.gz gzip: dict.xdb.gz: unexpected end of file还有一个scws-1.2.3.tar.gz\scws-1.2.3\dev\php4\pscws23_dictool\dict.txt可以用它生成字典。cws-gen-dict -h cws-gen-dict (scws-mkdict/1.2.3) Convert the plain text dictionary to xdb format. Copyright (C)2007 by hightman. Usage: scws-gen-dict [options] [input file] [output file] -i Specified the plain text dictionary(default: dict.txt). -o Specified the output file path(default: dict.xdb) -c Specified the input charset(default: gbk) -p Specified the PRIME num for xdb -v Show the version. -h Show this page. Report bugs to hightman2yahoo.com.cn scws-gen-dict -i scws-master/dict.txt Reading the input file: scws-master/dict.txt ...OK, total nodes346742 Optimizing... OK Dump the tree data to: dict.xdb ... OK, all been done!但是用它分词出来的还都是全英文数字scws -c utf8 -i 155.txt -t 80 -d dict.xdb No. WordString Attr Weight(times) ------------------------------------------------- 01. 2035 en 10.40(3) 02. PM2 en 8.24(3) ... 44. 8% en 1.73(1)还是从官方网站下载的比较安全。curl -LO http://www.xunsearch.com/scws/down/scws-dict-chs-utf8.tar.bz2 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 162 100 162 0 0 493 0 --:--:-- --:--:-- --:--:-- 493 100 3994k 100 3994k 0 0 4406k 0 --:--:-- --:--:-- --:--:-- 8052k curl -LO http://www.xunsearch.com/scws/down/scws-dict-chs-gbk.tar.bz2 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 162 100 162 0 0 2280 0 --:--:-- --:--:-- --:--:-- 2314 100 3936k 100 3936k 0 0 6588k 0 --:--:-- --:--:-- --:--:-- 6588k tar xvjf scws-dict-chs-gbk.tar.bz2 dict.xdb tar xvjf scws-dict-chs-utf8.tar.bz2 dict.utf8.xdb3.注意用匹配字符集的字典执行scws -c utf8 -i 155.txt -t 8 -d dict.utf8.xdb -M 1 No. WordString Attr Weight(times) ------------------------------------------------- 01. 发展 vn 1975.79(445) 02. 推进 v 1397.28(284) 03. 建设 vn 1336.50(297) 04. 加强 v 1268.46(261) 05. 完善 v 1212.78(246) 06. 体系 n 1200.50(245) 07. 健全 v 1162.80(204) 08. 机制 n 986.00(200) --[scws(scws-cli/1.2.3)]---------- | TextLen: 181431 | | Prepare: 0.0001 (sec) | | Segment: 0.0562 (sec) | -------------------------------- scws -c gbk -i 155gbk.txt -t 8 -d dict.xdb -M 1 No. WordString Attr Weight(times) ------------------------------------------------- 01. չ vn 1975.79(445) 02. ƽ v 1397.28(284) 03. vn 1336.50(297) 04. ǿ v 1268.46(261) 05. v 1212.78(246) 06. ϵ n 1200.50(245) 07. ȫ v 1162.80(204) 08. n 986.00(200) --[scws(scws-cli/1.2.3)]---------- | TextLen: 121696 | | Prepare: 0.0001 (sec) | | Segment: 0.0554 (sec) | --------------------------------分词结果完全相同只不过gbk代码的字体显示异常。