Parsing text file to csv in python

Your two examples have very different fields, but you could still create a CSV containing all fields you need as follows:

from itertools import groupby, takewhile, ifilter
import re
import csv

heading = None
sub_headings = ['Identifiers', 'Organism']
attribute_fields = []

# First scan to determine list of all used attribute_fields
with open('projects.txt') as f_projects:
    re_attributes = re.compile(r'    \/(.*?)=".*"')

    for line in f_projects:
        # '    /sample size="0.1, gram"'
        re_attribute = re_attributes.match(line)

        if re_attribute:
            attribute_fields.append(re_attribute.group(1))

# Remove duplicate attributes, sort and prefix the top fields
attribute_fields = ['Description', 'id', 'Accession', 'AccessionID'] + sorted(set(attribute_fields))    

with open('projects.txt') as f_projects, open('output.csv', 'wb') as f_output:
    csv_output = csv.DictWriter(f_output, fieldnames=sub_headings + attribute_fields)
    csv_output.writeheader()

    skip_empty_lines = ifilter(lambda x: len(x.strip()), f_projects)

    for k, v in groupby(skip_empty_lines, lambda x: re.match('\d+: ', x)):
        if k:
            heading = next(v).strip()
        elif heading:
            row = {'id' : heading}
            lines = list(v)

            for line_number, line in enumerate(lines):
                for sub_heading in sub_headings:
                    if line.startswith(sub_heading):
                        row[sub_heading] = line.split(':', 1)[1].strip()

                if line.startswith('Attributes:'):
                    for attribute in takewhile(lambda x: x.startswith('    /'), iter(lines[line_number+1:])):
                        k, v = re.findall(r'/(.*?)="(.*?)"', attribute)[0]
                        row[k] = v

                if line.startswith('Description:'): 
                    row['Description'] = lines[line_number+2].strip() # use next line only

                # Accession: SAMN00030407\tID: 30407
                if line.startswith('Accession:'):
                    accession, accession_id = re.match('Accession: (.*?)\tID: (.*?)$', line).groups()
                    row.update({'Accession':accession, 'AccessionID':accession_id})

            csv_output.writerow(row)

This would generate a rather sparse output CSV as follows:

Identifiers,Organism,Description,id,Accession,AccessionID,!16S_BarcodeSequence,"""PUBLIC""",16S_ForwardPrimer,16S_LinkerPrimerSequence,ArrayExpress-Species,ENA-CHECKLIST,ENA-FIRST-PUBLIC,ENA-LAST-UPDATE,HCA_MBT,HEIGHT,ITS2_BarcodeSequence,ITS2_LinkerPrimerSequence,PUBLIC,PlatformChemistry,Species,TOTAL_SCCA,WEIGHT,age,age at fmt,age group,age unit,age_unit,altitude,analyte type,anonymized name,anonymized_name,antibiotics,assigned from geo,assigned_from_geo,barcoded primer name,barcoded_primer_name,barcodesequence,bcs,bcs grouping,biomaterial provider,biospecimen repository,biospecimen repository sample id,body habitat,body mass index,body product,breed,breed grouping,calprotectin,cd behavior,cd location,cd resection,chemical administration,collection date,collection timestamp,common name,common_name,crude fiber 1000kcalg me group,cultivar,day since fmt,depth,description,detail,dewormed,diagnosis full,disease,disease control,dna extracted,donor group,donor kind,donor or patient,donor_recipient,ecotype,elevation,emp status,env matter,env_matter,environment biome,environment feature,environment material,environmental package,ethnicity,exp code,experiment center,experiment design description,experiment title,fecal date,fmt modality,g fat 1000kcal me group,g protein 1000kcal me group,gastrointestinal tract disorder,gender specific,geographic location,geographic location (country and/or sea),has extracted data,has physical specimen,health state,histo,histological type,hospitalized for fmt,host,host age,host body mass index,host body product,host disease,host disease status,host family relationship,host genotype,host sex,host subject id,host taxonomy ID,host tissue sampled,host-associated environmental package,human gut environmental package,ibd,ibd or not,ibd subtype,illumina technology,immune_state,immunocompromized,indiv g fat 1000kcal me group,indiv g protein 1000kcal me group,individual,indoor outdoor,inflammed,investigation type,isolate,isolation and growth condition,isolation source,lane,latitude,latitude (raw),latitude and longitude,library construction protocol,linker,linkerprimersequence,longitude,longitude (raw),marital status,mid,miscellaneous parameter,molecular data type,mouse_number,non barcoded linker,non barcoded primer,non barcoded primer name,non_barcoded_linker,non_barcoded_primer,non_barcoded_primer_name,number courses metronidazole,number fidaxo courses,number ivig,number prior episodes,number prior fmt,number recurrence after fmt,number std vanco courses,number vanco tapers,pathology,patient,patientnumber,pcr primers,pcr_primers,perc crude protein min group,percent crude fat min group,percent crude fiber max group,percent met cal carb group,percent met cal fat group,percent met cal protein group,perianal disease,perturbation,phenotype,physical location,physical specimen location,physical specimen remaining,platform,platformchemistry,pool name,postfmt cdi result,postfmt symptoms,prebotic source,primer plate,project name,protein source,public,race code,replicate,required sample info status,run center,run date,run prefix,sample center,sample collection device or method,sample id,sample name,sample no ngs nr,sample plate,sample size,sample storage temperature,sample type,sample_code,sample_id,sampling_time,secondary description,separate first,separate first and donor,seq_meth,sequencing meth,sequencing method,sex,size grouping,source material identifiers,state,state us,strain,study,study center,study design,study id,study name,subject,subject code,submitted sample id,submitted subject id,submitter handle,target gene,target subfragment,target_gene,target_subfragment,taxon id,taxon_id,terminal ileum,time_point,time_point_label,time_point_months,timepoint,tissue,title,total mass,travel history,treatment_parasite,uc extent,unknown,vanc plus rif chaser,visit_num,weight group,weight kg,well id,year diagnosed
BioSample: SAMEA3914960,Homo sapiens,Accession: SAMEA3914960    ID: 5788191,1: IBD22; ANALYSIS AND CHARACTERISATION OF THE FAECAL MICROBIAL DEGRADOME IN INFLAMMATORY BOWEL DISEASE,SAMEA3914960,5788191,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,2011,,,,,,,,,,,,,,,,,,,,,,,,Intestine,Colon,Faecal,,,,,,,,,,,,,,United Kingdom,,,,,,,,,,Faeces,,Inflammatory Bowel Disease,,,,,,,,human-gut,,,,,,,,,,,,metagenome,,,,,,51??31'03.3,,,,,,0??10'25.2,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,IBD gut,,,,,,,,,,,,ERS1102094,,,,,,,,,,,,,,Illumina Miseq,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
BioSample: SAMEA3914951; SRA: ERS1102085,Homo sapiens,Accession: SAMEA3914951   ID: 5788190,2: IBD13; ANALYSIS AND CHARACTERISATION OF THE FAECAL MICROBIAL DEGRADOME IN INFLAMMATORY BOWEL DISEASE,SAMEA3914951,5788190,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,2011,,,,,,,,,,,,,,,,,,,,,,,,Intestine,Colon,Faecal,,,,,,,,,,,,,,United Kingdom,,,,,,,,,,Faeces,,Inflammatory Bowel Disease,,,,,,,,human-gut,,,,,,,,,,,,metagenome,,,,,,51??31'03.3,,,,,,0??10'25.2,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,IBD gut,,,,,,,,,,,,ERS1102085,,,,,,,,,,,,,,Illumina Miseq,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

Tested on Python 2.7.12