Блог

Я передаю данные (байты) в firehose в формате JSON

Пример данных:

 b'{"itemid": 9763019456, "name": "(Beleza Cabelo) Tg Afro Kinky Curly Extens\\u00e3o Do Cabelo Rabo De Cavalo Para As Mulheres Com Cord\\u00e3o Sint\\u00e9tico Rabo De Cavalo Ombre Cor Borgonha Afro Kinky Curly Peda\\u00e7os De Cabelo Rabo De Cavalo", "shopid": 433979244, "status": 1, "ctime": 1622776968, "currency": "BRL", "historical_sold": 57, "sold": 46, "stock": 20940, "view_count": 6630, "liked_count": 377, "catid": 22359, "price": 4250000, "discount": "17%", "shopee_verified": false, "is_official_shop": false, "show_free_shipping": false, "is_preferred_plus_seller": false, "shop_location": "China Continental", "is_on_flash_sale": false, "is_cc_installment_payment_eligible": false, "spl_installment_tenure": "", "is_mart": false, "coin_info": "", "foody_item": "", "adsid": ""}'
 

И определение схемы в каталоге клея как:

И включение преобразования JSON в firehose, но это приводит к ошибке преобразования файла, сохранения этого файла в моем S3:

 {"attemptsMade":1,"arrivalTimestamp":1632234605915,"lastErrorCode":"DataFormatConversion.MalformedData","lastErrorMessage":"Data does not match the schema. For input string: "9763019456"","attemptEndingTimestamp":1632234911549,"rawData":"eyJpdGVtaWQiOiA5NzYzMDE5NDU2LCAibmFtZSI6ICIoQmVsZXphIENhYmVsbykgVGcgQWZybyBLaW5reSBDdXJseSBFeHRlbnNcXHUwMGUzbyBEbyBDYWJlbG8gUmFibyBEZSBDYXZhbG8gUGFyYSBBcyBNdWxoZXJlcyBDb20gQ29yZFxcdTAwZTNvIFNpbnRcXHUwMGU5dGljbyBSYWJvIERlIENhdmFsbyBPbWJyZSBDb3IgQm9yZ29uaGEgQWZybyBLaW5reSBDdXJseSBQZWRhXFx1MDBlN29zIERlIENhYmVsbyBSYWJvIERlIENhdmFsbyIsICJzaG9waWQiOiA0MzM5NzkyNDQsICJzdGF0dXMiOiAxLCAiY3RpbWUiOiAxNjIyNzc2OTY4LCAiY3VycmVuY3kiOiAiQlJMIiwgImhpc3RvcmljYWxfc29sZCI6IDU3LCAic29sZCI6IDQ2LCAic3RvY2siOiAyMDk0MCwgInZpZXdfY291bnQiOiA2NjMwLCAibGlrZWRfY291bnQiOiAzNzcsICJjYXRpZCI6IDIyMzU5LCAicHJpY2UiOiA0MjUwMDAwLCAiZGlzY291bnQiOiAiMTclIiwgInNob3BlZV92ZXJpZmllZCI6IGZhbHNlLCAiaXNfb2ZmaWNpYWxfc2hvcCI6IGZhbHNlLCAic2hvd19mcmVlX3NoaXBwaW5nIjogZmFsc2UsICJpc19wcmVmZXJyZWRfcGx1c19zZWxsZXIiOiBmYWxzZSwgInNob3BfbG9jYXRpb24iOiAiQ2hpbmEgQ29udGluZW50YWwiLCAiaXNfb25fZmxhc2hfc2FsZSI6IGZhbHNlLCAiaXNfY2NfaW5zdGFsbG1lbnRfcGF5bWVudF9lbGlnaWJsZSI6IGZhbHNlLCAic3BsX2luc3RhbGxtZW50X3RlbnVyZSI6ICIiLCAiaXNfbWFydCI6IGZhbHNlLCAiY29pbl9pbmZvIjogIiIsICJmb29keV9pdGVtIjogIiIsICJhZHNpZCI6ICIifQ==","sequenceNumber":"49622256859821865810714050835036460152140438080701071362","subSequenceNumber":null,"dataCatalogTable":{"catalogId":null,"databaseName":"scrapper","tableName":"shopee_produtos","region":"us-east-1","versionId":"LATEST","roleArn":"arn:aws:iam::894502253452:role/service-role/KinesisFirehoseServiceRole-PUT-shopee-fi-us-east-1-1632231040630"}}