Gloss-Based Pipeline for Spoken to Signed Language Translation

a text-to-gloss-to-pose-to-video pipeline for spoken to signed language translation.

• Demos available for:

  • 🇩🇪 Swiss German Sign Language 🇨🇭
  • 🇫🇷 French Sign Language of Switzerland🇨🇭
  • 🇮🇹 Italian Sign Language of Switzerland 🇨🇭

• Paper available on arxiv, presented at AT4SSL 2023.

Install

pip install spoken-to-signed

Then, to download a lexicon, run:

download_lexicon \
  --name <signsuisse> \
  --directory <path_to_directory>

Usage

For language codes, we use the IANA Language Subtag Registry. Our pipeline provides multiple scripts.

To quickly demo it using a dummy lexicon, either open it in Colab:

or run it locally with some installation steps first:

git clone https://github.com/ZurichNLP/spoken-to-signed-translation
cd spoken-to-signed-translation

pip install .

text_to_gloss_to_pose \
  --text "Kleine Kinder essen Pizza in Zürich." \
  --glosser "simple" \
  --lexicon "assets/dummy_lexicon" \
  --spoken-language "de" \
  --signed-language "sgg" \
  --pose "quick_test.pose"

Text-to-Gloss Translation

This script translates input text into gloss notation.

text_to_gloss \
  --text <input_text> \
  --glosser <simple|spacylemma|rules|nmt> \
  --spoken-language <de|fr|it> \
  --signed-language <sgg|ssr|slf>

Text-to-Gloss-to-Pose Translation

This script translates input text into gloss notation, then converts the glosses into a pose file.

text_to_gloss_to_pose \
  --text <input_text> \
  --glosser <simple|spacylemma|rules|nmt> \
  --lexicon <path_to_directory> \
  --spoken-language <de|fr|it> \
  --signed-language <sgg|ssr|slf> \
  --pose <output_pose_file_path>.pose

Text-to-Gloss-to-Pose-to-Video Translation

This script translates input text into gloss notation, converts the glosses into a pose file, and then transforms the pose file into a video.

Note: Video generation requires the pose-to-video package with pix2pix and upscaler:

pip install 'pose-to-video[pix2pix,simple_upscaler] @ git+https://github.com/sign-language-processing/pose-to-video'

text_to_gloss_to_pose_to_video \
  --text <input_text> \
  --glosser <simple|spacylemma|rules|nmt> \
  --lexicon <path_to_directory> \
  --spoken-language <de|fr|it> \
  --signed-language <sgg|ssr|slf> \
  --video <output_video_file_path>.mp4

Methodology

The pipeline consists of three main components:

1. Text-to-Gloss Translation

   Transforms the input (spoken language) text into a sequence of glosses.

• Simple lemmatizer,
• Spacy lemmatizer: more accurate, but slower lemmatization, covering fewer languages than simple,
• Rule-based word reordering and dropping component and
• Neural machine translation system.

2. Gloss-to-Pose Conversion

• Lookup: Uses a lexicon of signed languages to convert the sequence of glosses into a sequence of poses.
• Pose Concatenation: The poses are then cropped, concatenated, and smoothed, creating a pose representation for the input sentence.

3. Pose-to-Video Generation

   Transforms the processed pose video back into a synthesized video using an image translation model.

Supported Languages

Language	IANA Code	Glossers Supported	Lexicon Data Source
Swiss German Sign Language	sgg	simple, spacylemma, rules, nmt	SignSuisse (de)
Swiss French Sign Language	ssr	simple, spacylemma	SignSuisse (fr)
Swiss Italian Sign Language	slf	simple, spacylemma	SignSuisse (it)
German Sign Language	gsg	simple, spacylemma, nmt	WordNet (Coming Soon)
British Sign Language	bfi	simple, spacylemma, nmt	WordNet (Coming Soon)

Online Playgrounds

We have two available:

• sign.mt is a web interface of a translation system.
• research.sign.mt is an overview of sign language processing literature.

Citation

If you find this work useful, please cite our paper:

@inproceedings{moryossef2023baseline,
  title={An Open-Source Gloss-Based Baseline for Spoken to Signed Language Translation},
  author={Moryossef, Amit and M{\"u}ller, Mathias and G{\"o}hring, Anne and Jiang, Zifan and Goldberg, Yoav and Ebling, Sarah},
  booktitle={2nd International Workshop on Automatic Translation for Signed and Spoken Languages (AT4SSL)},
  year={2023},
  month={June},
  url={https://github.com/ZurichNLP/spoken-to-signed-translation},
  note={Available at: \url{https://arxiv.org/abs/2305.17714}}
}