Блог

Я пытаюсь запустить классификацию Huggingface с нулевым выстрелом для 200 строк, которые я хочу сопоставить с 20 категориями.

Требуется почти 10 ГБ оперативной памяти, но используется только один процессор. Вот что я сделал до сих пор:

 from transformers import pipeline

classifier = pipeline("zero-shot-classification")

labels = ["bread", "pasta", "cereal", "rice", "grains", "biscuits", 
              "crackers", "popcorn", "vegetables", "fruits", "meat", 
              "poultry", "fish", "eggs", "legumes", "nuts", "seeds", 
              "salad", "soup", "dessert", "sweets", "beverages", "souce"]
unique_titles = ['Cappuccino',
             'Stove Top Stuffing Mix For Turkey (Kraft)',
             'Roasted Dark Turkey Meat',
             'Low Fat 2% Small Curd Cottage Cheese (Daisy)',
             'Rice Cereal (Gerber)',
             'Oranges',
             "Cheese Spinach Pie (Heba's Health Foods)",
             'Turkey Sausage Egg amp; Cheese Croissant (Wawa)',
             'Bananas',
             'Light Nonfat Yogurt (Great Value)',
             'Blueberries',
             'Strawberries',
             'Macaroni',
             'Chicken Breast',
             'Mushrooms',
             'Jamoca Almond Fudge Ice Cream (Baskin-Robbins)',
             'Boston Cream Donut (Wawa)',
             "Plain Bagel (Dunkin' Donuts)",
             'Fried Egg',
             'Bison amp; Beef Hot Dogs (Great Range)',
             'Mushrooms Pieces amp; Stems (Great Value)',
             'Cookie',
             'Penne',
             'Beef Sausage',
             'Egg Omelet or Scrambled Egg with Mushrooms',
             'Apples',
             'Peach',
             'Ramen Noodles with Chicken Flavor (35% Less Sodium) (Maruchan)',
             'Greek Nonfat Yogurt - Plain (Friendly Farms)',
             'French or Vienna Roll',
             'Sea Salt Carmel Frozen Yogurt (Kemps)',
             'Gluten Free Thin amp; Crispy Signature Pepperoni Pizza (Freschetta)',
             'Red Tomatoes',
             'Sliced Ham  (Regular, Approx. 11% Fat)',
             "Low Moisture Part-skim Mozzarella Light String Cheese (Trader Joe's)",
             'Low Moisture-Part Skim Mozzarella String Cheese (Crystal Farms)',
             'Red Table Wine',
             'Sly Red’s Seeded Ancient Grain Bread (Simply Nature)',
             'Rose Wine',
             'Bison Meat (Lean Only)',
             'White Rice',
             'Baked or Broiled Salmon',
             'Hazelnuts Filberts',
             'Dark Chocolate amp; Nuts (Choceur)',
             'Deli Select Pastrami (Hillshire Farm)',
             'Ciabatta Sandwich Rolls (Specially Selected)',
             'Deli Sliced Mozzarella (Happy Farms)',
             'Enlive (Ensure)',
             'Greek Yogurt (Kirkland Signature)',
             'Salmon',
             'High Protein Shake - Chocolate (Premier Nutrition)',
             'Tangerine',
             'Pastrami (Columbus Salame)',
             'Olive Tapenade Hummus (Little Salad Bar)',
             'Multigrain Tortilla Chips (Simply Nature)',
             'Fruit Smoothie Drink',
             'Sliced Mushrooms (Kroger)',
             'Mozzarella Shredded Cheese (Sargento)',
             'Cooked Egg White',
             'Spinach (Chopped or Leaf, Frozen)',
             'Turkey Bacon',
             'Non Dairy Chocolate Protein Shake (14 oz) (Muscle Milk)',
             "Great Northern Beans (Dakota's Pride)",
             'Tomato Sauce (Happy Harvest)',
             'Pickled Green Tomato',
             'Grain Free Cookie Bites Chocolate Chip (Simply Nature)',
             'Jumbo Bison Hot Dog (Body By Bison)',
             'Spinach',
             'Chocolate Low-Fat Protein Shake (Muscle Milk)',
             'Organic Yellow Corn Tortilla Chips (Simply Nature)',
             'Rich Chocolate Complete Protein Shake (Pure Protein)',
             'Ramen Noodle Soup - Beef Flavor (Maruchan)',
             'White Corn Tortilla Chips (Delish)',
             'Vegan Veggie Burger (Earth Grown)',
             'Apple',
             'Peaches',
             'B12 Gummies (Sundown Naturals)',
             'String Cheese Mozzarella Cheese Snacks (Sargento)',
             'Natural Mozzarella String Cheese (Kraft)',
             'Vanilla Milkshake (Pure Protein)',
             'Hard-Boiled Egg',
             'Chicken Breast (Skin Not Eaten)',
             '0% Plain Greek Yogurt (8 oz) (Chobani)',
             'Grapes (Red or Green, European Type Varieties Such As Thompson Seedless)',
             "Tzaziki (Heba's Health Foods)",
             "Ham, Egg White amp; Cheese Wake-Up Wrap (Dunkin' Donuts)",
             'Whole Wheat Waffles (Essential Everyday)',
             "Large Grade A Eggs (Eggland's Best)",
             'Turkey Bacon (Appleton Farms)',
             'Strawberry Milkshake (Pure Protein)',
             'Feta Cheese',
             'Egg',
             'Portuguese Rolls (Stop amp; Shop)',
             'Chicken Bacon Ranch Casserole',
             'Carrot Muffin',
             "Chunky Honey Roasted Ham with Potatoes Soup (Campbell's)",
             'Biscotti Cookie (Italian Sugar Cookie)',
             'Chewy Granola Bars - Chocolate Chip (Quaker)',
             'Maple amp; Brown Sugar Instant Oatmeal (Great Value)',
             'Walnuts',
             'Taco Salad',
             'Potato French Fries',
             "Boneless Chicken Wings (Chester's)",
             'Avocado (Calavo)',
             'Bacon',
             'Chicken Enchilada Soup (Signature Cafe)',
             'Shredded Hash Browns (Great Value)',
             'Soft White Roll',
             'Chicken amp; Apple Sausage (Aidells)',
             'Bell Peppers',
             'Mayonnaise',
             'Classics Milk Chocolate Hot Cocoa Mix (Swiss Miss)',
             "Seasoned Steak Bites (Schwan's)",
             'Potatoes (Flesh, with Salt, Boiled)',
             'Lettuce Salad with Assorted Vegetables',
             'Everything Bagel (Publix)',
             'Oven Roasted Deli Style Chicken Breast (Kroger)',
             'Waffle Potato Fries (Small) (Chick-fil-A)',
             'Chicken Nuggets (8 Count) (Chick-fil-A)',
             'Pasta with Meat Sauce',
             'Garlic Bread',
             "Kettle Cooked Mesquite BBQ Chips (Package) (Lay's)",
             'Low-Fat Vegetarian Black Bean Soup (Bowl) (Panera Bread)',
             'Sour Dough Bread',
             'Snickerdoodle Cookie (Betty Crocker)',
             'Steak Sandwich on Roll',
             'Cauliflower Bites (Don Lee Farms)',
             'Black Bean Soup',
             'All Natural Creamy Almond Butter (Maranatha)',
             'Texas Toast (Wonder)',
             "Thin Crust Pizza (Nancy's Pizza)",
             'Saltine Crackers (Great Value)',
             'Chocolate Fudge Candies (with Nuts)',
             'Baguette',
             'Beef Stew with Potatoes and Vegetables in Gravy',
             'Chocolate Chip Ice Cream (Breyers)',
             'Pecan Nuts',
             'Hint of Lime Tortilla Chips (Tostitos)',
             "Chicken Enchilada Soup (Cup) (Chili's)",
             'Whole Wheat Bread',
             'pulled pork sandwich, regular',
             'Asian Chicken Salad (Large) (Urban Plates)',
             'Grilled Steak (Urban Plates)',
             'Tortilla Chips - Island Lime (Margaritaville)',
             'Oreo Thins',
             'Egg, Sausage, Cheese amp; Potato Breakfast Burrito (El Monterey)',
             'Apple Juice',
             'Frozen Blueberries',
             'Almond Milk Vanilla (Silk)',
             'Cashew Halves amp; Pieces (Great Value)',
             "Cocoa Special Dark (Hershey's)",
             'Pork Barbecue Sandwich or Sloppy Joe on Bun',
             'Sourdough Sliced Bread (San Luis Sourdough)',
             'Lettuce',
             'Organic Country-Style Potatoes (Cascadian Farm)',
             'Frozen Tropical Fruit Mix (Great Value)',
             'Pure Almond Milk - Original (Silk)',
             'Cashew Butter',
             'Organic Blue Agave Nectar (Wholesome Sweeteners)',
             'Stuffed Bell Peppers (Kirkland Signature)',
             'Taco Flavor Tortilla Chips (Doritos)',
             'Skinless Chicken Breast',
             'Young Green Onions',
             'Finely Shredded Mexican Style Four-cheese Blend Cheese (Market Pantry)',
             'Chocolate Covered Caramel with Nuts',
             'Garlic Naan Bread (Archer Farms)',
             'Red Baby Potatoes (Safeway)',
             'Grilled Bruschetta Chicken (Dream Dinners)',
             'Hashed Brown Potatoes',
             'Egg Roll in a Bowl',
             'Toasted Bagel',
             'Irish Soda Bread (Jewel)',
             'Chicken Barbecue Sandwich',
             'Low Carb Pancakes',
             'Garlic Naan (Tandoor Chef)',
             'Large Hamburger',
             'Garlic Fries (Gordon Biersch)',
             'Pork Egg Roll in A Bowl',
             'Chicken Nachos (Queso) (Personal) (Taco Cabana)',
             '100% Whole Wheat Bread',
             'Burrito with Chicken',
             "Tortilla Chips Hint of Lime (Clancy's)",
             'Baked Potato (Peel Eaten)',
             'Steak',
             "21 Whole Grains amp; Seeds Bread (Dave's Killer Bread)",
             'Clif Kid Organic Z Bar - Chocolate Chip (Clif Bar)',
             "Steak Street Taco (Rubio's Fresh Mexican Grill)",
             'Hamburger (Single Patty with Condiments)',
             'Ice Cream',
             'Tortilla Corn Chips',
             'Organic Brown Eggs (Large) (Organic Valley)',
             'Whole Wheat Bread (Commercial)',
             'Energy Granola (Clif Bar)',
             '33% Milk Chocolate (Chocolove)',
             'Baked Restructured Chips White Potato',
             'Shrimp',
             'Zucchini',
             'Cream Cheese',
             'Butter',
             'Toasted Garlic Bread']
        
classifier_dicts = classifier(unique_titles[0:100], labels, multi_class=True)
 

Пожалуйста, посоветуйте, как я могу распараллелить процесс, чтобы он выполнялся быстрее?