Блог

Системная информация

• Платформа и дистрибутив ОС: Windows 10
• Версия TensorFlow: 2.6.0
• Версия Python: 3.9
• Модель графического процессора и память: RTX 2060

Проблема

Я пытаюсь построить и обучить модель обнаружения объектов (используя ssd_resnet50_v1_fpn), настраивающую конвейер с путями для train.tfrecord, test.tfrecord и label_map.pbtxt. Я получаю последние файлы из cvat.org при экспорте проекта в формате TFRecord.

Информация / журналы

Я бегу: python model_main_tf2.py --model_dir=models/my_ssd_resnet50_v1_fpn --pipeline_config_path=models/my_ssd_resnet50_v1_fpn/pipeline.config

и ниже то, что я читаю:

 to use the following CPU instructions in performance-critical operations: AVX AVX2 To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2021-10-26 10:36:04.117867: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 3967 MB memory: -gt; device: 0, name: NVIDIA GeForce RTX 2060, pci bus id: 0000:01:00.0, compute capability: 7.5 INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0',) I1026 10:36:04.304352 16668 mirrored_strategy.py:369] Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0',) INFO:tensorflow:Maybe overwriting train_steps: None I1026 10:36:04.309246 16668 config_util.py:552] Maybe overwriting train_steps: None INFO:tensorflow:Maybe overwriting use_bfloat16: False I1026 10:36:04.309246 16668 config_util.py:552] Maybe overwriting use_bfloat16: False WARNING:tensorflow:From C:Usersdonatanaconda3envsdoEnvlibsite-packagesobject_detectionmodel_lib_v2.py:557: StrategyBase.experimental_distribute_datasets_from function (from tensorflow.python.distribute.distribute_lib) is deprecated and will be removed in a future version. Instructions for updating: rename to distribute_datasets_from_function W1026 10:36:04.327197 16668 deprecation.py:339] From C:Usersdonatanaconda3envsdoEnvlibsite-packagesobject_detectionmodel_lib_v2.py:557: StrategyBase.experime ntal_distribute_datasets_from_function (from tensorflow.python.distribute.distribute_lib) is deprecated and will be removed in a future version. Instructions for updating: rename to distribute_datasets_from_function INFO:tensorflow:Reading unweighted datasets: ['annotationsTrain.tfrecord'] I1026 10:36:04.334176 16668 dataset_builder.py:163] Reading unweighted datasets: ['annotationsTrain.tfrecord'] INFO:tensorflow:Reading record datasets for input file: ['annotationsTrain.tfrecord'] I1026 10:36:04.646700 16668 dataset_builder.py:80] Reading record datasets for input file: ['annotationsTrain.tfrecord'] INFO:tensorflow:Number of filenames to read: 1 I1026 10:36:04.646700 16668 dataset_builder.py:81] Number of filenames to read: 1 WARNING:tensorflow:num_readers has been reduced to 1 to match input file shards. W1026 10:36:04.648099 16668 dataset_builder.py:87] num_readers has been reduced to 1 to match input file shards. WARNING:tensorflow:From C:Usersdonatanaconda3envsdoEnvlibsite-packagesobject_detectionbuildersdataset_builder.py:101: parallel_interleave (from tensorflow.p ython.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.AUTOTUNE) instead. If sloppy execution is desired, use tf.data.Opti ons.experimental_deterministic. W1026 10:36:04.654085 16668 deprecation.py:339] From C:Usersdonatanaconda3envsdoEnvlibsite-packagesobject_detectionbuildersdataset_builder.py:101: parallel interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.AUTOTUNE) instead. If sloppy execution is desired, use tf.data.Opti ons.experimental_deterministic. WARNING:tensorflow:From C:Usersdonatanaconda3envsdoEnvlibsite-packagesobject_detectionbuildersdataset_builder.py:236: DatasetV1.map_with_legacy_function (fr om tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.data.Dataset.map() W1026 10:36:04.688991 16668 deprecation.py:339] From C:Usersdonatanaconda3envsdoEnvlibsite-packagesobject_detectionbuildersdataset_builder.py:236: DatasetV1 .map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.data.Dataset.map() WARNING:tensorflow:From C:Usersdonatanaconda3envsdoEnvlibsite-packagestensorflowpythonutildispatch.py:206: sparse_to_dense (from tensorflow.python.ops.spar se_ops) is deprecated and will be removed in a future version. Instructions for updating: Create a tf.sparse.SparseTensor and use tf.sparse.to_dense instead. W1026 10:36:10.687601 16668 deprecation.py:339] From C:Usersdonatanaconda3envsdoEnvlibsite-packagestensorflowpythonutildispatch.py:206: sparse_to_dense (fr om tensorflow.python.ops.sparse_ops) is deprecated and will be removed in a future version. Instructions for updating: Create a tf.sparse.SparseTensor and use tf.sparse.to_dense instead. WARNING:tensorflow:From C:Usersdonatanaconda3envsdoEnvlibsite-packagestensorflowpythonutildispatch.py:206: sample_distorted_bounding_box (from tensorflow.p ython.ops.image_ops_impl) is deprecated and will be removed in a future version. Instructions for updating: seed2 arg is deprecated.Use sample_distorted_bounding_box_v2 instead. W1026 10:36:13.237039 16668 deprecation.py:339] From C:Usersdonatanaconda3envsdoEnvlibsite-packagestensorflowpythonutildispatch.py:206: sample_distorted_bo unding_box (from tensorflow.python.ops.image_ops_impl) is deprecated and will be removed in a future version. Instructions for updating: seed2 arg is deprecated.Use sample_distorted_bounding_box_v2 instead. WARNING:tensorflow:From C:Usersdonatanaconda3envsdoEnvlibsite-packagestensorflowpythonautographimplapi.py:464: to_float (from tensorflow.python.ops.math_o ps) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. W1026 10:36:14.707938 16668 deprecation.py:339] From C:Usersdonatanaconda3envsdoEnvlibsite-packagestensorflowpythonautographimplapi.py:464: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. 2021-10-26 10:36:16.954501: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2) Traceback (most recent call last): File "D:programmitensorflowworkspacetraining_demomodel_main_tf2.py", line 115, in tf.compat.v1.app.run() File "C:Usersdonatanaconda3envsdoEnvlibsite-packagestensorflowpythonplatformapp.py", line 40, in run _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef) File "C:Usersdonatanaconda3envsdoEnvlibsite-packagesabslapp.py", line 303, in run _run_main(main, args) File "C:Usersdonatanaconda3envsdoEnvlibsite-packagesabslapp.py", line 251, in _run_main sys.exit(main(argv)) File "D:programmitensorflowworkspacetraining_demomodel_main_tf2.py", line 106, in main model_lib_v2.train_loop( File "C:Usersdonatanaconda3envsdoEnvlibsite-packagesobject_detectionmodel_lib_v2.py", line 599, in train_loop load_fine_tune_checkpoint( File "C:Usersdonatanaconda3envsdoEnvlibsite-packagesobject_detectionmodel_lib_v2.py", line 394, in load_fine_tune_checkpoint _ensure_model_is_built(model, input_dataset, unpad_groundtruth_tensors) File "C:Usersdonatanaconda3envsdoEnvlibsite-packagesobject_detectionmodel_lib_v2.py", line 159, in _ensure_model_is_built features, labels = iter(input_dataset).next() File "C:Usersdonatanaconda3envsdoEnvlibsite-packagestensorflowpythondistributeinput_lib.py", line 689, in next return self.next() File "C:Usersdonatanaconda3envsdoEnvlibsite-packagestensorflowpythondistributeinput_lib.py", line 693, in next return self.get_next() File "C:Usersdonatanaconda3envsdoEnvlibsite-packagestensorflowpythondistributeinput_lib.py", line 731, in get_next self._iterators[i].get_next_as_list_static_shapes(new_name)) File "C:Usersdonatanaconda3envsdoEnvlibsite-packagestensorflowpythondistributeinput_lib.py", line 1951, in get_next_as_list_static_shapes return self._format_data_list_with_options(self._iterator.get_next()) File "C:Usersdonatanaconda3envsdoEnvlibsite-packagestensorflowpythondataopsmulti_device_iterator_ops.py", line 573, in get_next result.append(self._device_iterators[i].get_next()) File "C:Usersdonatanaconda3envsdoEnvlibsite-packagestensorflowpythondataopsiterator_ops.py", line 814, in get_next return self._next_internal() File "C:Usersdonatanaconda3envsdoEnvlibsite-packagestensorflowpythondataopsiterator_ops.py", line 744, in _next_internal ret = gen_dataset_ops.iterator_get_next( File "C:Usersdonatanaconda3envsdoEnvlibsite-packagestensorflowpythonopsgen_dataset_ops.py", line 2727, in iterator_get_next _ops.raise_from_not_ok_status(e, name) File "C:Usersdonatanaconda3envsdoEnvlibsite-packagestensorflowpythonframeworkops.py", line 6941, in raise_from_not_ok_status six.raise_from(core._status_to_exception(e.code, message), None) File "", line 3, in raise_from tensorflow.python.framework.errors_impl.InvalidArgumentError: Input is empty. [[{{node case/cond/else/_10/case/cond/cond_jpeg/else/_105/case/cond/cond_jpeg/decode_image/DecodeImage}}]] [[MultiDeviceIteratorGetNextFromShard]] [[RemoteCall]] [Op:IteratorGetNext] ```