[hts-users:04124] parallel makefile for data in slt demo (2.2)
Hello
I've rewritten the data/Makefile.in from slt-based hts-demo (version 2.2) so that it can use:
1. the "-j" parameter of make to run several jobs in parallel
2. the automatic templates with dependencies to avoid re-running computations uselessly
it's been written as a drop-in replacement of data/Makefile.in, all the main targets are the same (all, labels, analysis, mgc, lf0, cmp, scp, list, mlf). This makes some parts inefficient, but not significantly.
I tested it on two different hardwares (laptop and cluster) with two different Linux distributions (ubuntu 12.04 and centOS 6.3) and didn't run into any error (doesn't mean it's bullet proof though)
As a rule of thumb, it should be run as:
> $ make -jnum_of_cores data
and, as a result, the computation time for the data target should be reduced by a factor close to num_of_cores, give or take (I couldn't test i7 CPUs for "make -jnum_of_core*2" but it might just work ...).
As an example, on a 24-core cluster, the computation time for "make data" (thus using 1 core) was a bit more than 1h30 minutes whereas "make -j24 data" takes about 6 minutes (so, an overall improvement factor of 15. The "cmp" target is accelerated by more than 20 but the gain for the "labels" target is much smaller, around 5, I guess it has to do with it being text- and script-based)
Besides, re-running the make command a second time should only compute the files that were either not completed in the first run (e.g. in case the script was stopped it will re-start where it was stopped) or were modified in-between (e.g. if one modifies utts/cmu_us_arctic_slt_a0001.utt, make will only recompute the label for that file, same thing for the raw/mgc/lf0 files to create the cmp). mlf, scp and list are still recreated each time, but it takes only seconds. Use "make clean" to force a full re-run of the computation.
main drawback: it looks like it doesn't work with BSD make (tested with FreeBSD 10). Using GNU make (gmake) from ports bypasses the problem for now.
I'm aware that most of the time in the demo is spent in the training part (make voice), but I hope it will help some people nonetheless.
Alexis
ps: please feel free to share any fix and improvement, some parts are a bit hackish ;-)
pps: it shouldn't be too hard to port to 2.3
# ----------------------------------------------------------------- #
# The HMM-Based Speech Synthesis System (HTS) #
# developed by HTS Working Group #
# http://hts.sp.nitech.ac.jp/ #
# ----------------------------------------------------------------- #
# #
# Copyright (c) 2001-2011 Nagoya Institute of Technology #
# Department of Computer Science #
# #
# 2001-2008 Tokyo Institute of Technology #
# Interdisciplinary Graduate School of #
# Science and Engineering #
# #
# 2014 Numediart Institute #
# Department of Signal Processing #
# #
# All rights reserved. #
# #
# Redistribution and use in source and binary forms, with or #
# without modification, are permitted provided that the following #
# conditions are met: #
# #
# - Redistributions of source code must retain the above copyright #
# notice, this list of conditions and the following disclaimer. #
# - Redistributions in binary form must reproduce the above #
# copyright notice, this list of conditions and the following #
# disclaimer in the documentation and/or other materials provided #
# with the distribution. #
# - Neither the name of the HTS working group nor the names of its #
# contributors may be used to endorse or promote products derived #
# from this software without specific prior written permission. #
# #
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND #
# CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, #
# INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF #
# MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE #
# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS #
# BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, #
# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED #
# TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, #
# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON #
# ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, #
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY #
# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE #
# POSSIBILITY OF SUCH DAMAGE. #
# ----------------------------------------------------------------- #
# setting
SPEAKER = @SPEAKER@
DATASET = @DATASET@
ARRAYDATASET = @ARRAYDATASET@
# awk, perl, bc and tcl
AWK = @AWK@
PERL = @PERL@
BC = @BC@
TCLSH = @TCLSH@
WC = @WC@
# SPTK commands
X2X = @X2X@
FRAME = @FRAME@
WINDOW = @WINDOW@
MGCEP = @MGCEP@
LPC2LSP = @LPC2LSP@
STEP = @STEP@
MERGE = @MERGE@
VSTAT = @VSTAT@
NRAND = @NRAND@
SOPR = @SOPR@
VOPR = @VOPR@
NAN = @NAN@
MINMAX = @MINMAX@
# dumpfeats to extract utterance information
DUMPFEATS = @DUMPFEATS@
# speech analysis conditions
SAMPFREQ = @SAMPFREQ@ # Sampling frequency (48kHz)
FRAMELEN = @FRAMELEN@ # Frame length in point (1200 = 48000 * 0.025)
FRAMESHIFT = @FRAMESHIFT@ # Frame shift in point (240 = 48000 * 0.005)
WINDOWTYPE = @WINDOWTYPE@ # Window type -> 0: Blackman 1: Hamming 2: Hanning
NORMALIZE = @NORMALIZE@ # Normalization -> 0: none 1: by power 2: by magnitude
FFTLEN = @FFTLEN@ # FFT length in point
FREQWARP = @FREQWARP@ # frequency warping factor
GAMMA = @GAMMA@ # pole/zero weight for mel-generalized cepstral (MGC) analysis
MGCORDER = @MGCORDER@ # order of MGC analysis
LNGAIN = @LNGAIN@ # use logarithmic gain rather than linear gain
LOWERF0 = @LOWERF0@ # lower limit for f0 extraction (Hz)
UPPERF0 = @UPPERF0@ # upper limit for f0 extraction (Hz)
NOISEMASK = @NOISEMASK@ # standard deviation of white noise to mask noises in f0 extraction
# windows for calculating delta features
MGCWIN = win/mgc.win
LF0WIN = win/lf0.win
NMGCWIN = @NMGCWIN@
NLF0WIN = @NLF0WIN@
RAW_FILES = $(wildcard raw/$(DATASET)_$(SPEAKER)_*.raw)
MGC_FILES = $(patsubst raw/%.raw, mgc/%.mgc, $(RAW_FILES))
LF0_FILES = $(patsubst raw/%.raw, lf0/%.lf0, $(RAW_FILES))
CMP_FILES = $(patsubst raw/%.raw, cmp/%.cmp, $(RAW_FILES))
UTT_FILES = $(wildcard utts/$(DATASET)_$(SPEAKER)_*.utt)
MONO_LAB_FILES = $(patsubst utts/%.utt, labels/mono/%.lab, $(UTT_FILES))
FULL_LAB_FILES = $(patsubst utts/%.utt, labels/full/%.lab, $(UTT_FILES))
all: analysis labels
analysis: mgc lf0 cmp
labels: label mlf list scp
test:
for i in $(ARRAYDATASET); do \
echo $${i} ; \
done
mgc: $(MGC_FILES)
lf0: $(LF0_FILES)
cmp: $(CMP_FILES)
mgc-dir:
@if [ ! -d "mgc" ]; then echo "create directory mgc" && mkdir -p mgc; fi
lf0-dir:
@if [ ! -d "lf0" ]; then echo "create directory lf0" && mkdir -p lf0; fi
cmp-dir:
@if [ ! -d "cmp" ]; then echo "create directory cmp" && mkdir -p cmp; fi
label: mono full
mono: $(MONO_LAB_FILES)
full: $(FULL_LAB_FILES)
labels/mono:
@if [ ! -d "labels/mono" ]; then echo "create directory labels/mono" && mkdir -p labels/mono; fi
labels/full:
@if [ ! -d "labels/full" ]; then echo "create directory labels/full" && mkdir -p labels/full; fi
lists-dir:
@if [ ! -d "lists" ]; then echo "create directory lists" && mkdir -p lists; fi
scp-dir:
@if [ ! -d "scp" ]; then echo "create directory scp" && mkdir -p scp; fi
tmp:
@if [ ! -d "tmp" ]; then echo "create directory tmp" && mkdir -p tmp; fi
mgc/%.mgc: raw/%.raw | mgc-dir
@min=`$(X2X) +sf $< | $(MINMAX) | $(X2X) +fa | head -n 1`; \
max=`$(X2X) +sf $< | $(MINMAX) | $(X2X) +fa | tail -n 1`; \
if [ -s $< -a $${min} -gt -32768 -a $${max} -lt 32767 ]; then \
$(X2X) +sf $< > tmp_$*_mgc; \
if [ $(GAMMA) -eq 0 ]; then \
echo "Extracting MGC coefficients from $<"; \
$(FRAME) -l $(FRAMELEN) -p $(FRAMESHIFT) tmp_$*_mgc | \
$(WINDOW) -l $(FRAMELEN) -L $(FFTLEN) -w $(WINDOWTYPE) -n $(NORMALIZE) | \
$(MGCEP) -a $(FREQWARP) -m $(MGCORDER) -l $(FFTLEN) -e 1.0E-08 > $@; \
else \
echo "Extracting MGC-LSP coefficients from $<"; \
SAMPKHZ=`expr $(SAMPFREQ) / 1000`; \
if [ $(LNGAIN) -eq 1 ]; then \
GAINOPT="-l"; \
fi; \
$(FRAME) -l $(FRAMELEN) -p $(FRAMESHIFT) tmp_$*_mgc | \
$(WINDOW) -l $(FRAMELEN) -L $(FFTLEN) -w $(WINDOWTYPE) -n $(NORMALIZE) | \
$(MGCEP) -a $(FREQWARP) -c $(GAMMA) -m $(MGCORDER) -l $(FFTLEN) -e 1.0E-08 -o 4 | \
$(LPC2LSP) -m $(MGCORDER) -s $${SAMPKHZ} $${GAINOPT} -n $(FFTLEN) -p 8 -d 1.0E-08 > $@; \
fi; \
rm -f tmp_$*_mgc; \
if [ -n "`$(NAN) $@`" ]; then \
echo " Failed to extract MGC coefficients from $<"; \
rm -f $@; \
fi; \
fi;
lf0/%.lf0: raw/%.raw | lf0-dir
@min=`$(X2X) +sf $< | $(MINMAX) | $(X2X) +fa | head -n 1`; \
max=`$(X2X) +sf $< | $(MINMAX) | $(X2X) +fa | tail -n 1`; \
if [ -s $< -a $${min} -gt -32768 -a $${max} -lt 32767 ]; then \
echo "Extracting f0 from $<"; \
count=`echo "0.005 * $(SAMPFREQ)" | $(BC) -l`; \
$(STEP) -l `printf "%.0f" $${count}` -v 0.0 | \
$(X2X) +fs > tmp_$*.head; \
count=`echo "0.025 * $(SAMPFREQ)" | $(BC) -l`; \
$(STEP) -l `printf "%.0f" $${count}` -v 0.0 | \
$(X2X) +fs > tmp_$*.tail; \
cat tmp_$*.head $< tmp_$*.tail | \
$(X2X) +sf > tmp_$*_lf0; \
leng=`$(X2X) +fa tmp_$*_lf0 | $(WC) -l`; \
$(NRAND) -l $${leng} | $(SOPR) -m $(NOISEMASK) | $(VOPR) -a tmp_$*_lf0 | \
$(X2X) +fs > tmp_$*.raw; \
$(TCLSH) scripts/getf0.tcl -l -lf0 -H $(UPPERF0) -L $(LOWERF0) -p $(FRAMESHIFT) -r $(SAMPFREQ) tmp_$*.raw | \
$(X2X) +af > $@; \
rm -f tmp_$**; \
if [ -n "`$(NAN) $@`" ]; then \
echo " Failed to extract log f0 from $<"; \
rm -f $@; \
fi; \
fi;
cmp/%.cmp: mgc/%.mgc lf0/%.lf0 | cmp-dir
@MGCDIM=`expr $(MGCORDER) + 1`; \
LF0DIM=1; \
MGCWINDIM=`expr $(NMGCWIN) \* $${MGCDIM}`; \
LF0WINDIM=`expr $(NLF0WIN) \* $${LF0DIM}`; \
BYTEPERFRAME=`expr 4 \* \( $${MGCWINDIM} + $${LF0WINDIM} \)`; \
MGCWINS=""; \
LF0WINS=""; \
if [ -s $< -a -s lf0/$*.lf0 ]; then \
i=1; \
echo "Composing training data for $*"; \
while [ $${i} -le $(NMGCWIN) ]; do \
eval MGCWINS=\"$${MGCWINS} $(MGCWIN)$${i}\"; \
i=`expr $${i} + 1`; \
done; \
$(PERL) scripts/window.pl $${MGCDIM} $< $${MGCWINS} > cmp/tmp_$*.mgc; \
i=1; \
while [ $${i} -le $(NLF0WIN) ]; do \
eval LF0WINS=\"$${LF0WINS} $(LF0WIN)$${i}\"; \
i=`expr $${i} + 1`; \
done; \
$(PERL) scripts/window.pl $${LF0DIM} lf0/$*.lf0 $${LF0WINS} > cmp/tmp_$*.lf0; \
$(MERGE) +f -s 0 -l $${LF0WINDIM} -L $${MGCWINDIM} cmp/tmp_$*.mgc < cmp/tmp_$*.lf0 > cmp/tmp_$*.cmp; \
$(PERL) scripts/addhtkheader.pl $(SAMPFREQ) $(FRAMESHIFT) $${BYTEPERFRAME} 9 cmp/tmp_$*.cmp > $@; \
rm -f cmp/tmp_$*.*; \
fi;
tmp/tmp_%.lab: utts/%.utt | tmp
@echo analyzing $<
@if [ -s $< ]; then \
$(DUMPFEATS) \
-eval scripts/extra_feats.scm \
-relation Segment \
-feats scripts/label.feats \
-output $@ \
$<; \
fi;
labels/mono/%.lab: tmp/tmp_%.lab | labels/mono
@echo creating $@
@$(AWK) -f scripts/label-mono.awk $< > $@;
labels/full/%.lab: tmp/tmp_%.lab | labels/full
@echo creating $@
@$(AWK) -f scripts/label-full.awk $< > $@;
mlf:
@echo "Generating monophone Master Label Files (MLF)"
@echo "#!MLF!#" > labels/mono.mlf
@echo "\"*/$(DATASET)_$(SPEAKER)_*.lab\" -> \"@PWD@/data/labels/mono\"" >> labels/mono.mlf
@echo "Generating fullcontext Master Label Files (MLF)"
@echo "#!MLF!#" > labels/full.mlf
@echo "\"*/$(DATASET)_$(SPEAKER)_*.lab\" -> \"@PWD@/data/labels/full\"" >> labels/full.mlf
list: cmp label lists-dir
@echo "Generating a fullcontext model list file"; \
rm -f tmp_list; \
for lab in labels/full/$(DATASET)_$(SPEAKER)_*.lab; do \
if [ -s $${lab} -a -s labels/mono/`basename $${lab}` -a -s cmp/`basename $${lab} .lab`.cmp ]; then \
sed -e "s/.* //g" $${lab} >> tmp_list; \
fi \
done; \
sort -u tmp_list > lists/full.list; \
rm -f tmp_list
@echo "Generating a fullcontext model list file which includes unseen models"; \
rm -f tmp_list; \
cat lists/full.list > tmp_list; \
for lab in labels/gen/*.lab; do \
sed -e "s/.* //g" $${lab} >> tmp_list; \
done; \
sort -u tmp_list > lists/full_all.list; \
rm -f tmp_list
@echo "Generating a monophone model list file"; \
rm -f tmp_list; \
for lab in labels/mono/$(DATASET)_$(SPEAKER)_*.lab; do \
if [ -s $${lab} -a -s labels/full/`basename $${lab}` -a -s cmp/`basename $${lab} .lab`.cmp ]; then \
sed -e "s/.* //g" $${lab} >> tmp_list; \
fi \
done; \
sort -u tmp_list > lists/mono.list; \
rm -f tmp_list
scp: cmp label scp-dir
@echo "Generating a training data script"; \
rm -f scp/train.scp; \
for cmp in @PWD@/data/cmp/$(DATASET)_$(SPEAKER)_*.cmp; do \
if [ -s $${cmp} -a -s labels/mono/`basename $${cmp} .cmp`.lab -a -s labels/full/`basename $${cmp} .cmp`.lab ]; then \
echo $${cmp} >> scp/train.scp; \
fi \
done;
@echo "Generating a generation label script"; \
rm -f scp/gen.scp; \
for lab in @PWD@/data/labels/gen/*.lab; do \
echo $${lab} >> scp/gen.scp; \
done
clean: clean-mgc clean-lf0 clean-cmp clean-label clean-mlf clean-list clean-scp clean-tmp
clean-mgc:
rm -rf mgc
clean-lf0:
rm -rf lf0
clean-cmp:
rm -rf cmp
clean-label:
rm -rf labels/mono
rm -rf labels/full
clean-mlf:
rm -f labels/*.mlf
clean-list:
rm -rf lists
clean-scp:
rm -rf scp
clean-tmp:
rm -rf tmp
distclean: clean
rm -f Makefile
.PHONY: all analysis labels mgc lf0 cmp label mlf list scp clean
.PRECIOUS: lf0/%.lf0 mgc/%.mgc cmp/%.cmp labels/mono/%.lab labels/full/%.lab
- Follow-Ups
-
- [hts-users:04125] Re: parallel makefile for data in slt demo (2.2), Matt Shannon