Java 类名：com.alibaba.alink.operator.batch.dataproc.HugeStringIndexerPredictBatchOp
Python 类名：HugeStringIndexerPredictBatchOp

功能介绍

提供字符串ID化处理功能，与 StringIndexerPredictBatchOp 功能相同，是其升级版本，模型为分布式存储，提升了运行效率。支持多列同时转换。
由 StringIndexerTrainBatchOp 生成词典模型，将输入数据的字符串转化成词典模型中的ID
对于词典模型中不存在的字符串，提供了三种处理策略，”keep”表示用最大id加1代替, “skip”表示补null， “error”表示抛异常

参数说明

名称 中文名称 描述 类型 是否必须？ 取值范围 默认值
selectedCols 选择的列名 计算列对应的列名列表 String[] ✓
handleInvalid 未知token处理策略 未知token处理策略。”keep”表示用最大id加1代替, “skip”表示补null， “error”表示抛异常 String “KEEP”, “ERROR”, “SKIP” “KEEP”
outputCols 输出结果列列名数组 输出结果列列名数组，可选，默认null String[] null
reservedCols 算法保留列名 算法保留列 String[] null

代码示例

Python 代码

  1. from pyalink.alink import *
  3. import pandas as pd
  5. useLocalEnv(1)
  7. df = pd.DataFrame([
  8.     ["football", "apple"],
  9.     ["football", "apple"],
  10.     ["football", "apple"],
  11.     ["basketball", "apple"],
  12.     ["basketball", "apple"],
  13.     ["tennis", "pair"],
  14.     ["tennis", "pair"],
  15.     ["pingpang", "banana"],
  16.     ["pingpang", "banana"],
  17.     ["baseball", "banana"]
  18. ])
  20. data = BatchOperator.fromDataframe(df, schemaStr='f0 string, f1 string')
  22. stringindexer = StringIndexerTrainBatchOp()\
  23.     .setSelectedCol("f0")\
  24.     .setSelectedCols(["f1"])\
  25.     .setStringOrderType("alphabet_asc")
  27. model = stringindexer.linkFrom(data)
  29. predictor = HugeStringIndexerPredictBatchOp()\
  30.     .setSelectedCols(["f0", "f1"])\
  31.     .setOutputCols(["f0_indexed", "f1_indexed"])
  33. predictor.linkFrom(model, data).print()

Java 代码

  1. import org.apache.flink.types.Row;
  3. import com.alibaba.alink.operator.batch.BatchOperator;
  4. import com.alibaba.alink.operator.batch.source.MemSourceBatchOp;
  5. import org.junit.Test;
  7. import java.util.Arrays;
  8. import java.util.List;
  10. public class HugeStringIndexerPredictBatchOpTest {
  11.     @Test
  12.     public void testStringIndexerPredictBatchOp() throws Exception {
  13.         List <Row> df = Arrays.asList(
  14.             Row.of("football", "apple"),
  15.             Row.of("football", "apple"),
  16.             Row.of("football", "apple"),
  17.             Row.of("basketball", "apple"),
  18.             Row.of("basketball", "apple"),
  19.             Row.of("tennis", "pair"),
  20.             Row.of("tennis", "pair"),
  21.             Row.of("pingpang", "banana"),
  22.             Row.of("pingpang", "banana"),
  23.             Row.of("baseball", "banana")
  24.         );
  25.         BatchOperator <?> data = new MemSourceBatchOp(df, "f0 string,f1 string");
  26.         BatchOperator <?> stringindexer = new StringIndexerTrainBatchOp()
  27.             .setSelectedCol("f0")
  28.             .setSelectedCols("f1")
  29.             .setStringOrderType("frequency_asc");
  30.         BatchOperator <?> predictor = new HugeStringIndexerPredictBatchOp().setSelectedCols("f0", "f1")
  31.             .setOutputCols("f0_indexed", "f1_indexed");
  32.         BatchOperator model = stringindexer.linkFrom(data);
  33.         model.lazyPrint(10);
  34.         BatchOperator result = predictor.linkFrom(model, data);
  35.         result.print();
  36.     }
  37. }

运行结果

| token | token_index | | —- | —- |

| banana | 5 |

| football | 6 |

| basketball | 1 |

| pingpang | 2 |

| tennis | 3 |

| pair | 4 |

| baseball | 0 |

| apple | 7 |

| f0 | f1 | f0_indexed | f1_indexed | | —- | —- | —- | —- |

| basketball | apple | 1 | 7 |

| pingpang | banana | 2 | 5 |

| football | apple | 6 | 7 |

| tennis | pair | 3 | 4 |

| tennis | pair | 3 | 4 |

| basketball | apple | 1 | 7 |

| football | apple | 6 | 7 |

| football | apple | 6 | 7 |

| pingpang | banana | 2 | 5 |

| baseball | banana | 0 | 5 |