Блог

Привет, сообщество Data science, я новичок в data science и программировании на python. Вот структура моего txt-файла, но в нем много пропущенных значений

 #*Improved Channel Routing by Via Minimization and Shifting.
#@Chung-Kuan Cheng
David N. Deutsch
#t1988
#cDAC
#index131751
#3716
#3521
#4343
#!Channel routing area improvement by means of via minimization and via shifting in two dimensions (compaction) is readily achievable. Routing feature area can be minimized by wire straightening. The implementation of algorithms for each of these procedures has produced a solution for Deutsch's Difficult Example
 the standard channel routing benchmark
 that is more than 5% smaller than the best result published heretofore. Suggestions for possible future work are also given.

#*A fast simultaneous input vector generation and gate replacement algorithm for leakage power reduction.
#@Lei Cheng
Liang Deng
Deming Chen
Martin D. F. Wong
#t2006
#cDAC
#index131752
#2550
#S0568
#C6486
#4259
#(3007
#4422
#(2140
#34324
#!Input vector control (IVC) technique is based on the observation that the leakage current in a CMOS logic gate depends on the gate input state
 and a good input vector is able to minimize the leakage when the circuit is in the sleep mode. The gate replacement technique is a very effective method to further reduce the leakage current. In this paper
 we propose a fast algorithm to find a low leakage input vector with simultaneous gate replacement. Results on MCNC91 benchmark circuits show that our algorithm produces $14 %$ better leakage current reduction with several orders of magnitude speedup in runtime for large circuits compared to the previous state-of-the-art algorithm. In particular
 the average runtime for the ten largest combinational circuits has been dramatically reduced from 1879 seconds to 0.34 seconds.

#*On the Over-Specification Problem in Sequential ATPG Algorithms.
#@Kwang-Ting Cheng
Hi-Keung Tony Ma
#t1992
#cDAC
#index131756
#E5537
#78626
#1745
#!The authors show that some ATPG (automatic test pattern generation) programs may err in identifying untestable faults. These test generators may not be able to find the test sequence for a testable fault
 even allowed infinite run time
 and may mistakenly claim it as untestable. The main problem of these programs is that the underlying combinational test generation algorithm may over-specify the requirements at the present state lines. A necessary condition that the underlying combinational test generation algorithm must satisfy is considered to ensure a correct sequential ATPG program. It is shown that the simple D-algorithm satisfies this condition while PODEM and the enhanced D-algorithm do not. The impact of over-specification on the length of the generated test sequence was studied. Over-specification caused a longer test sequence. Experimental results are presented

#*Device and architecture co-optimization for FPGA power reduction.
#@Lerong Cheng
Phoebe Wong
Fei Li
Yan Lin
Lei He
#t2005
#cDAC
#index131759
#!4244
#!5701
#!4503
#(2575
#!4411
#!4505
#2929
#!Device optimization considering supply voltage Vdd and threshold voltage Vt tuning does not increase chip area but has a great impact on power and performance in the nanometer technology. This paper studies the simultaneous evaluation of device and architecture optimization for FPGA. We first develop an efficient yet accurate timing and power evaluation method
 called trace-based model. By collecting trace information from cycle-accurate simulation of placed and routed FPGA benchmark circuits and re-using the trace for different Vdd and Vt
 we enable the device and architecture co-optimization for hundreds of combinations. Compared to the baseline FPGA which has the architecture same as the commercial FPGA used by Xilinx
 and has Vdd suggested by ITRS but Vt optimized by our device optimization
 architecture and device co-optimization can reduce energy-delay product by 20.5% without any chip area increase compared to the conventional FPGA architecture. Furthermore
 considering power-gating of unused logic blocks and interconnect switches
 our co-optimization method reduces energy-delay product by 54.7% and chip area by 8.3%. To the best of our knowledge
 this is the first in-depth study on architecture and device co-optimization for FPGAs.

  

Я хочу преобразовать его во фрейм данных, строки exp, которые начинаются с #@, являются авторами, #! являются аннотациями, # * — заголовками, #% — ссылками, а # c — местами, использующими python
Каждая статья начинается с ее названия, проблема может быть связана с рефератами

Я пробовал разные подходы, такие как

 import csv
with open('names7.csv', 'w', encoding="utf-8") as csvfile:
    fieldnames = ["Venue", "Year", "Authors","Title","id","ListCitation","NbrCitations","Abstract"]
    writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
    writer.writeheader()
    with open(r"C:UserslenovoDownloads1.txt", "r", encoding="utf-8") as f:
        cnt = 1
        for line in f :           
            if line.startswith('#*'):
                writer.writerow({'Title': line})
                cnt  = 1
            elif line.startswith('#@'):
                writer.writerow({'Authors': line})
                cnt  = 1
            elif line.startswith("#t"):
                writer.writerow({'Year': line})
                cnt  = 1
            elif line.startswith("#!"):
                writer.writerow({'Abstract': line})
                cnt  = 1
            elif line.startswith("#c"):
                writer.writerow({'Venue': line})
                cnt  =1
            elif line.startswith("#index"):
                writer.writerow({'id': line}) 
                cnt  =1
            else:
                writer.writerow({'ListCitation': line}) 
                cnt  =1
    f.close()
  

Я попробовал этот подход, но он не сработал, я хочу преобразовать его в фрейм данных с указанными столбцами, как я могу преобразовать этот файл в фрейм данных и сохранить результат в файле csv

вывод кода ответа

Там вывод, который я хочу

например, для этого случая (абстрактный столбец) между абзацами есть пробел, который вызвал у меня проблему, поэтому этот случай необходимо учитывать, а для ссылки на столбец есть много ссылок, поэтому их нужно учитывать

 #*Total power reduction in CMOS circuits via gate sizing and multiple threshold voltages.
#@Feng Gao
John P. Hayes
#t2005
#cDAC
#index132139
#C7038
#C7006
#C6596
#(5977
#35402
#2206
#4016
#3061
#!Minimizing power consumption is one of the most important objectives in IC design. Resizing gates and assigning different Vt's are common ways to meet power and timing budgets. We propose an automatic implementation of both these techniques using a mixedinteger linear programming model called MLP-exact
 which minimizes a circuit's total active-mode power consumption. Unlike previous linear programming methods which only consider local optimality
 MLP-exact
 can find a true global optimum. An efficient
 non-optimal way to solve the MLP model
 called MLP-fast

 is also described. We present a set of benchmark experiments which show that MLP-fast
 is much faster than MLP-exact

 while obtaining designs with only slightly higher power consumption. Furthermore
 the designs generated by MLP-fast
 consume 30% less power than those obtained by conventional
 sensitivity-based methods.
  

csv.DictWriter().writerow() принимает словарь, представляющий всю строку для этого конкретного образца. Что происходит с вашим кодом прямо сейчас, так это то, что все, что вы вызываете writerow, создает совершенно новую строку вместо добавления к текущей строке. Вместо этого вам следует сделать следующее

1. определите переменную с именем row в качестве словаря
2. сохраните значения для всей строки в этой переменной
3. запись в csv с использованием writerow для этого словаря

Это приведет к записи всей строки в csv вместо новой строки для каждого нового значения.

Хотя это не единственная проблема, на которую мы должны обратить внимание. Каждая новая строка в текстовом документе, у которой нет начального идентификатора, считается значением, которым она не является. Например!Канал разделен на 3 строки текста, но только первая строка будет рассматриваться как !Channel , а остальные 2 строки будут рассматриваться как что-то другое.

Ниже приведена улучшенная версия кода с документацией, использующая словарь для хранения начальных значений и соответствующих идентификаторов. Чтобы добавить новые обращения, просто измените словарь keys и fieldnames

 """
#@ are authors
#! are abstracts
#* are titles
#% are references
#index are index
#c are venues using python Each article start by its title , the problem may have to do with abstracts
"""

# use dictionary to store fieldnames with corresponding id's/tags
keys = {
        'Venue': '#c',
        'Year':'#t',
        'Authors':'#@',
        'Title':'#*',
        'id': '#index',
        'References': '#%',
        'Abstract': '#!',
}

fieldnames = ["Venue", "Year", "Authors", "Title","id","Abstract", 'References']

outFile = 'names7.csv' # path to csv output
inFile = r"1.txt" # path to input text file

import csv
with open(outFile, 'w', encoding="utf-8") as csvfile:
    writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
    writer.writeheader()
    with open(inFile, "r", encoding="utf-8") as f:
        row = dict()
        stack = [] # (key, value) pair, used to store multiple values regarding rows
        for line in f:
            line = line.strip() # remove white space from beginning and end of line
            prev = "" # store value of previous column
            for col in fieldnames: 
                # if column is defined and doesnt start with any id, add to previous value
                # this handles cases with results over one line or containing new lines
                if prev in row and not any([line.startswith(prefix) for prefix in keys.items()]):
                    # remove prefix
                    prefix = keys[prev]
                    line = line[len(prefix):]

                    row[prev]  = ' '   line
                # initate or append to current value. Handles (References #%)
                elif col in keys and line.startswith(keys[col]):
                    # remove prefix
                    prefix = keys[col]
                    line = line[len(prefix):]
                    if col in row:
                        stack.append((col, line))
                    else:
                        row[col] = line
                    prev = col # define prev col if answer goes over one line
                    break # go to next line in text
        writer.writerow(row)
        for col, line in stack:
            row[col] = line
            writer.writerow(row)
        f.close()
  

Результат, полученный в приведенном выше тестовом примере.

Обновлен предыдущий ответ с этим результатом, полученным с учетом этого конкретного текстового файла

 """
#@ are authors
#! are abstracts
#* are titles
#% are references
#index are index
#c are venues using python Each article start by its title , the problem may have to do with abstracts
"""

# use dictionary to store fieldnames with corresponding id's/tags
keys = {
    '#c': 'Venue',
    '#t': 'Year',
    '#@': 'Authors',
    '#*': 'Title',
    '#index': 'id',
    '#%': 'References',
    '#!': 'Abstract'
}


fieldnames = ["Venue", "Year", "Authors", "Title", "NbrAuthor", "id", "ListCitation", "NbrCitation", "References", "NbrReferences", "Abstract"]

# fieldnames = ["Venue", "Year", "Authors", "Title", "NbrAuthor", "id", "ListCitation", "NbrCitation"]



# References and Authors store on one line
# Count number of authors and references
# We want to count the Authors, NbrAuthor, NbrCitations

outFile = 'names7.csv' # path to csv output
inFile = r"1.txt" # path to input text file

import csv
import re
with open(outFile, 'w', encoding="utf-8") as csvfile:
    writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
    writer.writeheader()
    with open(inFile, "r", encoding="utf-8") as f:
        row = dict()
        prev = ""
        for line in f.readlines():
            line = line.strip() # remove any leading or trailing whitespace
            # remove any parentheses at the end of the string
            query = re.findall(r'([^)]*)', line)
            if len(query) > 0:
                line = line.replace(query[-1], '')
            # if none of the keys match, then belongs to previous key
            if prev != "" and not any([line.startswith(k) for k in keys]):
                if prev == 'Abstract':
                    row[prev]  = " "   line
                else:
                    row[prev]  = ", "   line
            else:
                for k in keys:
                    prefix = ""
                    if line.startswith(k):
                        # remove prefix
                        prefix = k
                        line = line[len(prefix):]
                        if keys[k] in row:
                            if keys[k] == "References":
                                row[keys[k]]  = ", "   line
                            else:
                                row[keys[k]]  = " "   line
                        else:
                            row[keys[k]] = line
                        prev = keys[k]
        # count number of references and Citations
        row["NbrAuthor"] = row["Authors"].count(',')   1
        row["NbrCitation"] = 0
        row["NbrReferences"] = row["References"].count(',')   1
        writer.writerow(row)
  

Редактировать: добавлено предложение в оператор if

 prefixes = {
    '#*': 'Title',
    '#@': 'Authors',
    '#t': 'Year',
    '#c': 'Venue',
    '#index': 'id',
    '#%': 'References',
    '#!': 'Abstract',
}


outFile = 'names7.csv' # path to csv output
inFile = r"1.txt" # path to input text file

import csv
import re
with open(outFile, 'w', encoding="utf-8") as csvfile:
    writer = csv.DictWriter(csvfile, fieldnames=list(prefixes.values())   ['NbrAuthor', 'NbrCitations', 'ListCitations'])
    writer.writeheader()
    with open(inFile, "r", encoding="utf-8") as f:
        row = dict()
        prev = ''
        for line in f.readlines():
            # remove leading and trailing whitespace
            line = line.strip()

            # remove close brackets at end of lines
            # query = re.findall(r'([^)]*)', line)
            # if len(query) > 0:
            #     line = line.replace(query[-1], '')

            for prefix, col in prefixes.items():
                if line.startswith(prefix):
                    line = line[len(prefix):]
                    if col == "Authors" or col == 'Abstract':
                        row[col] = ""
                    else:
                        row[col] = line
                    prev = prefix
                    break
            
            # special cases
            try:
                if prev == '#@':
                    if row['Authors'] == "":
                        row['Authors'] = line
                    else:
                        row['Authors']  = ', '   line
                elif prev == '#%':
                    row['References']  = ', '   line
                elif prev == '#!':
                    row['Abstract']  = ' '   line
            except Exception as e:
                print(e)
                
            if len(line) == 0:
                row['NbrAuthor'] = row['Authors'].count(',')   1
                row['NbrCitations'] = 0
                writer.writerow(row)
                prev = ''
                row = dict()