V2EX = way to explore
V2EX 是一个关于分享和探索的地方
Sign Up Now
For Existing Member  Sign In
dreamxj001
V2EX  ›  问与答

来个 doris 大神,咨询一下 ROUTINE LOAD 的问题

  •  
  •   dreamxj001 · Nov 26, 2024 · 1354 views
    This topic created in 517 days ago, the information mentioned may be changed or developed.
    • 场景: 从 kafka 消费原始报文,做简单的数据抽取,并把原文存储到 doris 表中

    • 表结构:

      1. k1, varchar
      2. k2, varchar
      3. packet, json
    • routine load 配置

    CREATE ROUTINE LOAD xxx.yyy
    ON zzz
    COLUMNS (
        k1 ,
        k2 ,
        packet
    )
    PROPERTIES
    (
        "format" = "json",
        "jsonpaths"="[\"$.k1\",\"$.k2\",\"$\"]",
        "max_batch_rows"="1000000",
        "max_batch_size"="104857600",
        "max_filter_ratio"="0.5",
        "max_error_number" = "1000",
        "strict_mode" = "true"
    )
    FROM KAFKA
    (
        "kafka_broker_list" = "xxx",
        "kafka_topic" = "xxx",
        "property.kafka_default_offsets" = "OFFSET_BEGINNING",
        "property.group.id" = "doris-routine"
    )
    COMMENT "报文导入";
    
    • 问题 ROUTINE LOAD 可以正常创建,但是运行起来后就会报错: Reason: [DATA_QUALITY_ERROR]empty json path. src line [];

    如果把 jsonpaths 改为"["$.k1","$.k2"]" 就能正常写入数据

    但是业务场景需要将 kafka 消息原文进行存储,请问大佬应该如何处理

    dreamxj001
        1
    dreamxj001  
    OP
       Nov 26, 2024
    消息报文结构是:
    dreamxj001
        2
    dreamxj001  
    OP
       Nov 26, 2024
    @dreamxj001 {"k1":"111", "k2":"222", "k3":"333",.....}
    About   ·   Help   ·   Advertise   ·   Blog   ·   API   ·   FAQ   ·   Solana   ·   2983 Online   Highest 6679   ·     Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 · 35ms · UTC 15:23 · PVG 23:23 · LAX 08:23 · JFK 11:23
    ♥ Do have faith in what you're doing.