Add the auto segmenter config to font2ift.

This allows font2ift to perform the full IFT encoding process:
1. Auto generate segmenter config.
2. Run segmenter.
3. Compile the font.

If a segmentation plan is not supplied to font2ift it will then using the segemnter auto config and closure segmenter to generate one.

Author: garretrieger

README.md

@@ -58,7 +58,7 @@ script:
 ## Documentation
 
 The documents under [docs/experimental](docs/experimental) provide some more detailed designs of various aspects of the IFT encoder. Of note:
-* [compiler.md](docs/experimental)
+* [compiler.md](docs/experimental/compiler.md)
 * [closure_glyph_segmentation.md](docs/experimental/closure_glyph_segmentation.md)
 * [closure_glyph_segmentation_merging.md](docs/experimental/closure_glyph_segmentation_merging.md)
 * [closure_glyph_segmentation_complex_conditions.md](docs/experimental/closure_glyph_segmentation_complex_conditions.md)
@@ -77,13 +77,81 @@ bazel run @hedron_compile_commands//:refresh_all
 
 Will generate a compile_commands.json file.
 
-## Producing IFT Encoded Fonts
+## Producing IFT Encoded Fonts (with Auto Config)
 
-IFT encoded fonts are produced in two steps:
-1. A segmentation plan is generated which specifies how the font file should be split up in the IFT encoding.
-2. The IFT encoded font and patches are compiled by the Compiler sub module using the segmentation plan.
+The simplest way to create IFT fonts is via the `font2ift` utility utilizing the auto configuration mode.
+This is done by running the utility and not providing a segmentation plan. Example invocation:
 
-### Generating Segmentation Plan
+```bash
+bazel run -c opt @ift_encoder//util:font2ift -- \
+  --input_font="$HOME/fonts/myfont/MyFont.ttf" \
+  --output_path=$HOME/fonts/myfont/ift/ \
+  --output_font="MyFont-IFT.woff2"
+```
+
+This will analyze the input font, decide how to segment it, and then produce the final IFT encoded font
+and patches.
+
+When utilizing auto config there are two optional flags which can be used to adjust the behaviour:
+* `--auto_config_primary_script`: this tells the config generator which language/script the font is intended
+  to be used with. It has two effects: first the codepoints of the primary script are eligible to be moved
+  into the initial font. Second for scripts with large overlaps, such as CJK, primary script selects which
+  of the overlapping scripts to use frequency data from. Values refer to frequency data files in
+  [ift-encoder-data](https://github.com/w3c/ift-encoder-data/tree/main/data). Example values: "Script_bengali",
+  "Language_fr"
+
+* `--auto_config_quality`: This is analagous to a quality level in a compression library. It controls how much
+  effort is spent to improve the efficiency of the final IFT font. Values range from 1 to 8, where higher
+  values increase encoding times but typically result in a more efficient end IFT font (ie. less bytes
+  transferred by clients using it).
+
+Example command line with optional flags:
+
+```bash
+bazel run -c opt @ift_encoder//util:font2ift -- \
+  --input_font="$HOME/fonts/NotoSansJP-Regular.otf" \
+  --output_path=$HOME/fonts/ift/ \
+  --output_font="NotoSansJP-Regular-IFT.woff2" \
+  --auto_config_primary_script=Script_japanese \
+  --auto_config_quality=3
+```
+
+*Note: the auto configuration mode is still under development, in particular the auto selection of quality level
+is currently quite simplistic. It's expected to continue to evolve from it's current state.*
+
+## Producing IFT Encoded Fonts (Advanced)
+
+Under the hood IFT font encoding happens in three stages:
+
+1. Generate or write a segmenter config for the font.
+2. Generate a segmentation plan, which describes how the font is split into patches. Takes the segmenter config as an input.
+3. Compile the final IFT encoded font following the segmentation plan.
+
+For more advanced use cases these steps can be performed individually. This allows the segmenter config
+and segmentation plans to be fine tuned beyond what auto configuration is capable of.
+
+### Step 1: Generating a Segmenter Config
+
+There are two main options for generating a segmenter config:
+
+1. Write the config by hand, the segmenter is configured via an input configuration file using the
+    [segmenter_config.proto](util/segmenter_config.proto) schema, see the comments there for more details.
+    This option is useful when maximum control over segmentation parameters is needed, or custom frequency
+    data is being supplied.
+
+2. Auto generate the segmenter config using `util:generate_segmenter_config`.
+
+   ```
+   CC=clang bazel run //util:generate_segmenter_config -- \
+     --quality=5 \
+     --input_font=$HOME/MyFont.ttf > config.txtpb
+   ```
+
+   This analyzes the input font and tries to pick appropriate config values automatically. As discussed in
+   the previous "Producing IFT Encoded Fonts" section there is a configurable quality level. If needed
+   the auto generated config can be hand tweaked after generation.
+
+### Step 2: Generating Segmentation Plan
 
 Segmentation plans are in a [textproto format](https://protobuf.dev/reference/protobuf/textformat-spec/) using the
 [segmentation_plan.proto](util/segmentation_plan.proto) schema. See the comments in the schema file for more information.
@@ -93,17 +161,9 @@ possible to write plans by hand, or develop new utilities to generate plans.
 
 In this repo 3 options are currently provided:
 
-1.  `util/generate_table_keyed_config`: this utility generates the table keyed (extension segments that augment non
-    glyph data in the font) portion of a plan. Example execution:
-
-    ```sh
-    bazel run -c opt util:generate_table_keyed_config -- \
-      --font=$(pwd)/myfont.ttf \
-      latin.txt cyrillic.txt greek.txt > table_keyed.txtpb
-    ```
-
-2.  `util/closure_glyph_keyed_segmenter_util`: this utility uses a subsetting closure based approach to generate a glyph
-    keyed segmentation plan (extension segments that augment glyph data). Example execution:
+1. [Recommended] `util/closure_glyph_keyed_segmenter_util`: this utility uses a subsetting closure based approach
+    to generate a glyph keyed segmentation plan (extension segments that augment glyph data). It can optionally
+    generate the table keyed portion of the config as well. Example execution:
 
     ```sh
     bazel run -c opt util:closure_glyph_keyed_segmenter_util  -- \
@@ -119,6 +179,15 @@ In this repo 3 options are currently provided:
     Note: this utility is under active development and still very experimental. See
     [the status section](docs/experimental/closure_glyph_segmentation.md#status) for more details.
 
+2.  `util/generate_table_keyed_config`: this utility generates the table keyed (extension segments that augment non
+    glyph data in the font) portion of a plan. Example execution:
+
+    ```sh
+    bazel run -c opt util:generate_table_keyed_config -- \
+      --font=$(pwd)/myfont.ttf \
+      latin.txt cyrillic.txt greek.txt > table_keyed.txtpb
+    ```
+
 3.  `util/iftb2config`: this utility converts a segmentation obtained from the
     [binned incremental font transfer prototype](https://github.com/adobe/binned-ift-reference)
     into and equivalent segmentation plan. Example execution:
@@ -128,23 +197,20 @@ In this repo 3 options are currently provided:
       bazel run util:iftb2config > segmentation_plan.txtpb
     ```
 
-If seperate glyph keyed and table keyed configs were generated using #1 and #2 they can then be combined into one
+If separate glyph keyed and table keyed configs were generated using #1 and #2 they can then be combined into one
 complete plan by concatenating them:
 
 ```sh
 cat glyph_keyed.txtpb table_keyed.txtpb > segmentation_plan.txtpb
 ```
 
-Additional tools for generating encoder configs are planned to be added in the future.
-
 For concrete examples of how to generate IFT fonts, see the [IFT Demo](https://github.com/garretrieger/ift-demo).
 In particular the [Makefile](https://github.com/garretrieger/ift-demo/blob/main/Makefile) and the
 [segmenter configs](https://github.com/garretrieger/ift-demo/tree/main/config) may be helpful.
 
-### Generating an IFT Encoding
+### Step 3: Generating an IFT Encoding
 
-Once an segmentation plan has been created it can be combined with the target font to produce and incremental font and collection
-of associated patches using the font2ift utility which is a wrapper around the compiler. Example execution:
+Once a segmentation plan has been created it can be combined with the target font to produce an incremental font and collection of associated patches using the font2ift utility which is a wrapper around the compiler. Example execution:
 
 ```sh
 bazel -c opt run util:font2ift  -- \

ift/encoder/closure_glyph_segmenter.cc

@@ -734,4 +734,37 @@ Status ClosureGlyphSegmenter::FallbackCost(
   return absl::OkStatus();
 }
 
+void ClosureGlyphSegmenter::AddTableKeyedSegments(
+    SegmentationPlan& plan,
+    const btree_map<SegmentSet, MergeStrategy>& merge_groups,
+    const std::vector<SubsetDefinition>& segments,
+    const SubsetDefinition& init_segment) {
+  std::vector<SubsetDefinition> table_keyed_segments;
+  for (const auto& [segment_ids, _] : merge_groups) {
+    SubsetDefinition new_segment;
+    for (uint32_t s : segment_ids) {
+      new_segment.Union(segments.at(s));
+    }
+    new_segment.Subtract(init_segment);
+    table_keyed_segments.push_back(new_segment);
+  }
+
+  uint32_t max_id = 0;
+  for (const auto& [id, _] : plan.segments()) {
+    if (id > max_id) {
+      max_id = id;
+    }
+  }
+
+  uint32_t next_id = max_id + 1;
+  auto* plan_segments = plan.mutable_segments();
+  for (const SubsetDefinition& def : table_keyed_segments) {
+    GlyphSegmentation::SubsetDefinitionToSegment(def,
+                                                 (*plan_segments)[next_id]);
+    SegmentsProto* segment_ids = plan.add_non_glyph_segments();
+    segment_ids->add_values(next_id);
+    next_id++;
+  }
+}
+
 }  // namespace ift::encoder

ift/encoder/closure_glyph_segmenter.h

@@ -4,13 +4,15 @@
 #include <optional>
 #include <vector>
 
+#include "absl/container/btree_map.h"
 #include "absl/status/statusor.h"
 #include "ift/encoder/glyph_segmentation.h"
 #include "ift/encoder/merge_strategy.h"
 #include "ift/encoder/segmentation_context.h"
 #include "ift/encoder/subset_definition.h"
 #include "ift/freq/probability_calculator.h"
 #include "util/common.pb.h"
+#include "util/segmentation_plan.pb.h"
 #include "util/segmenter_config.pb.h"
 
 namespace ift::encoder {
@@ -89,6 +91,12 @@ class ClosureGlyphSegmenter {
                             uint32_t& fallback_glyphs_size,
                             uint32_t& all_glyphs_size) const;
 
+  static void AddTableKeyedSegments(
+      SegmentationPlan& plan,
+      const absl::btree_map<common::SegmentSet, MergeStrategy>& merge_groups,
+      const std::vector<SubsetDefinition>& segments,
+      const SubsetDefinition& init_segment);
+
  private:
   uint32_t brotli_quality_;
   uint32_t init_font_merging_brotli_quality_;

util/BUILD

@@ -64,9 +64,15 @@ cc_binary(
     srcs = [
         "font2ift.cc",
     ],
+    data = [
+        "@ift_encoder_data//:freq_data",
+    ],
     deps = [
+        ":auto_config_flags",
+        ":auto_segmenter_config",
         ":load_codepoints",
         ":segmentation_plan_cc_proto",
+        ":segmenter_config_util",
         "//common",
         "//ift",
         "//ift/encoder",
@@ -76,6 +82,7 @@ cc_binary(
         "@abseil-cpp//absl/status:statusor",
         "@abseil-cpp//absl/strings",
         "@harfbuzz",
+        "//util:segmenter_config_cc_proto",
     ],
 )
 
@@ -103,6 +110,7 @@ cc_binary(
         "@ift_encoder_data//:freq_data",
     ],
     deps = [
+        ":auto_config_flags",
         ":auto_segmenter_config",
         ":load_codepoints",
         ":segmentation_plan_cc_proto",
@@ -138,6 +146,16 @@ cc_binary(
     ],
 )
 
+cc_library(
+    name = "auto_config_flags",
+    srcs = ["auto_config_flags.cc"],
+    hdrs = ["auto_config_flags.h"],
+    visibility = ["//visibility:public"],
+    deps = [
+        "@abseil-cpp//absl/flags:flag",
+    ],
+)
+
 cc_library(
     name = "convert_iftb",
     srcs = [
@@ -203,10 +221,12 @@ cc_library(
     ],
     deps = [
         ":load_codepoints",
+        ":segmentation_plan_cc_proto",
         ":segmenter_config_cc_proto",
         "//common",
         "//ift/encoder",
         "@abseil-cpp//absl/status:statusor",
+        "@harfbuzz",
     ],
 )
 

util/auto_config_flags.cc

@@ -0,0 +1,15 @@
+#include "util/auto_config_flags.h"
+
+#include <string>
+
+#include "absl/flags/flag.h"
+
+ABSL_FLAG(int, auto_config_quality, 0,
+          "The quality level to use when generating a segmenter config. A value of 0 "
+          "means auto pick. Valid values are 1-8.");
+
+ABSL_FLAG(std::string, auto_config_primary_script, "Script_latin",
+          "When auto_config is enabled this sets the primary script or "
+          "language frequency data file to use. "
+          "The primary script is eligible to have codepoints moved to the init font. "
+          "For CJK primary script can be used to specialize against a specific language/script.");

util/auto_config_flags.h

@@ -0,0 +1,11 @@
+#ifndef UTIL_AUTO_CONFIG_FLAGS_H_
+#define UTIL_AUTO_CONFIG_FLAGS_H_
+
+#include <string>
+
+#include "absl/flags/declare.h"
+
+ABSL_DECLARE_FLAG(int, auto_config_quality);
+ABSL_DECLARE_FLAG(std::string, auto_config_primary_script);
+
+#endif  // UTIL_AUTO_CONFIG_FLAGS_H_

util/auto_segmenter_config.cc

@@ -2,7 +2,6 @@
 
 #include <cctype>
 #include <string>
-#include <unordered_map>
 
 #include "absl/container/flat_hash_set.h"
 #include "absl/log/log.h"
@@ -50,12 +49,6 @@ enum Quality {
   MAX = 8, // Alias for EIGHT
 };
 
-// TODO(garretrieger): define a very basic set of quality levels first (see next TODO),
-//   start with just a lowest and highest to set the upper and lower bounds for quality
-//   settings (maybe also a mid point). To begin use number of codepoints to select quality
-//   level. Do some testing on segmentation times at low and high to get a sense of
-//   how times are impacted.
-
 // TODO(garretrieger): do something analagous to brotli quality levels
 // where we define a series of levels which correspond to a set of
 // values for the quality/performance tradeoff settings (including setting the
@@ -291,7 +284,7 @@ StatusOr<std::string> AutoSegmenterConfig::GetBaseScriptForLanguage(
   }
 
   static const auto* lang_to_script =
-      new std::unordered_map<std::string, std::string>{
+      new flat_hash_map<std::string, std::string> {
           {"Language_af", "Script_latin"},
           {"Language_ak", "Script_latin"},
           {"Language_am", "Script_ethiopic"},
@@ -602,7 +595,7 @@ static void ApplyQualityLevelTo(Quality quality, SegmenterConfig& config) {
   }
 }
 
-absl::StatusOr<SegmenterConfig> AutoSegmenterConfig::GenerateConfig(
+StatusOr<SegmenterConfig> AutoSegmenterConfig::GenerateConfig(
     hb_face_t* face, std::optional<std::string> primary_script, std::optional<int> quality_level) {
   SegmenterConfig config;
   config.set_generate_table_keyed_segments(true);
@@ -617,9 +610,22 @@ absl::StatusOr<SegmenterConfig> AutoSegmenterConfig::GenerateConfig(
   auto freq_list = TRY(BuiltInFrequenciesList());
   CodepointSet unicodes = FontHelper::ToCodepointsSet(face);
   uint32_t cp_count = unicodes.size();
-  Quality quality = cp_count > 2000 ? MIN : MAX;
-  if (quality_level.has_value() && quality_level.value() >= ONE && quality_level.value() <= MAX) {
+
+  // TODO(garretrieger): more sophisticated scheme for auto picking quality level.
+  // roughly we want to estimate the expected cost of each quality level and pick
+  // based on that.
+  Quality quality = THREE;
+  if (cp_count <= 1000) {
+    quality = MAX;
+  } else if (cp_count <= 3000) {
+    quality_level = SIX;
+  }
+
+  if (quality_level.has_value() && quality_level.value() >= MIN && quality_level.value() <= MAX) {
     quality = static_cast<Quality>(quality_level.value());
+    VLOG(0) << "Using specified quality level for segmenting: " << quality;
+  } else {
+    VLOG(0) << "Quality level unspecified, auto picked: " << quality;
   }
 
   // Detect scripts by intersection with frequency data
@@ -644,7 +650,6 @@ absl::StatusOr<SegmenterConfig> AutoSegmenterConfig::GenerateConfig(
 
     cost->set_built_in_freq_data_name(script);
     if (script == primary_script_file) {
-      // TODO(garretrieger): customize these values based on the quality level
       cost->set_initial_font_merge_threshold(-60);
     }
   }
@@ -654,4 +659,4 @@ absl::StatusOr<SegmenterConfig> AutoSegmenterConfig::GenerateConfig(
   return config;
 }
 
-}  // namespace util
+}  // namespace util