MOSCA issueshttps://gitlab.bio.di.uminho.pt/iquasere/MOSCA/-/issues2021-02-02T21:39:08Zhttps://gitlab.bio.di.uminho.pt/iquasere/MOSCA/-/issues/13No adapter removal despite Adapter Content Fail in FastQC report2021-02-02T21:39:08ZJoão SequeiraNo adapter removal despite Adapter Content Fail in FastQC report*Created by: nnordhol*
Hi,
Nice to see you were able to integrate the MT only-mode in 1.3.
The preprocessing would fail to recognise that adapters were present (as reported by FastQC) and skip adapter removal. I think I identified ...*Created by: nnordhol*
Hi,
Nice to see you were able to integrate the MT only-mode in 1.3.
The preprocessing would fail to recognise that adapters were present (as reported by FastQC) and skip adapter removal. I think I identified the problem in preprocess.py: the has_adapter() function checks for over represented sequences when it should check for "Adapter Content" in the FastQC report. Changing the function as below solved it for me and would be the correct way to identify whether adapters are present from FastQC report:
def has_adapters(self, fastqc_report):
data = parse_fastqc(fastqc_report)
if data['Adapter Content'][0] == 'pass':
return False
else:
return True
Best,
Niclas
https://gitlab.bio.di.uminho.pt/iquasere/MOSCA/-/issues/9Versioning / releases2021-02-25T14:45:23ZJoão SequeiraVersioning / releases*Created by: d4straub*
Hi there,
I am looking into metatranscriptomics workflows and found yours. One major drawback I see is that you have no releases/versioning and therefore unnecessarily make reproducibility harder.
I was also...*Created by: d4straub*
Hi there,
I am looking into metatranscriptomics workflows and found yours. One major drawback I see is that you have no releases/versioning and therefore unnecessarily make reproducibility harder.
I was also wondering whether it is possible to use your pipeline with an existing metagenome assembly & annotation, the readme implies that there need to be MG and MT reads, did I understand that correctly?
Best wisheshttps://gitlab.bio.di.uminho.pt/iquasere/MOSCA/-/issues/14No such file or directory: 'output/Preprocess/SortMeRNA/read1.txt'2021-03-09T02:48:31ZJoão SequeiraNo such file or directory: 'output/Preprocess/SortMeRNA/read1.txt'*Created by: szypanther*
hi iquasere,
Sorry for trouble you again, right now I testing more data by MOSCA.
During the process, I encounter such error but the pipeline not stop and still continue run.
Is this missing file wil...*Created by: szypanther*
hi iquasere,
Sorry for trouble you again, right now I testing more data by MOSCA.
During the process, I encounter such error but the pipeline not stop and still continue run.
Is this missing file will impact the whole process or we can ignore it?
............................
join output/Preprocess/SortMeRNA/read1.txt output/Preprocess/SortMeRNA/read2.txt | awk '{print $1" "$2"\n"$3"\n+\n"$4 > "output/Preprocess/SortMeRNA/mt3namerep_forward.fastq";print $1" "$5"\n"$6"\n+\n"$7 > "output/Preprocess/SortMeRNA/mt3namerep_reverse.fastq"}'
Traceback (most recent call last):
File "/media/zyshen/work/MOSCA/MOSCA-1.3.5/workflow/preprocess.py", line 379, in <module>
Preprocesser().run()
File "/media/zyshen/work/MOSCA/MOSCA-1.3.5/workflow/preprocess.py", line 349, in run
original_files=True if args.input == original_input else False)
File "/media/zyshen/work/MOSCA/MOSCA-1.3.5/workflow/preprocess.py", line 226, in rrna_removal
'{}/{}_reverse.fastq'.format(out_dir, name), out_dir)
File "/media/zyshen/work/MOSCA/MOSCA-1.3.5/workflow/preprocess.py", line 191, in remove_orphans
os.remove(file)
FileNotFoundError: [Errno 2] No such file or directory: 'output/Preprocess/SortMeRNA/read1.txt'
[Thu Mar 4 16:38:34 2021]
Error in rule preprocess:
jobid: 0
output: output/Preprocess/Trimmomatic/quality_trimmed_mt3name_forward_paired.fq, output/Preprocess/Trimmomatic/quality_trimmed_mt3name_reverse_paired.fq
RuleException:
CalledProcessError in line 111 of /media/zyshen/work/MOSCA/MOSCA-1.3.5/workflow/Snakefile:
Command 'set -euo pipefail; python /media/zyshen/work/MOSCA/MOSCA-1.3.5/workflow/preprocess.py -i /media/zyshen/MOSCA/20201023_L_QMK/mt3_R1.fastq,/media/zyshen/MOSCA/20201023_L_QMK/mt3_R2.fastq -t 10 -o output/Preprocess -adaptdir /media/zyshen/work/MOSCA/MOSCA-1.3.4/adapters -rrnadbs /media/zyshen/work/MOSCA/MOSCA-1.3.4/rRNA_databases -d mrna -rd /media/zyshen/work/MOSCA/MOSCA-1.3.4 -n mt3name --minlen 100 --avgqual 20' returned non-zero exit status 1.
File "/media/zyshen/miniconda3/envs/snakemake/lib/python3.6/site-packages/snakemake/executors/__init__.py", line 2340, in run_wrapper
File "/media/zyshen/work/MOSCA/MOSCA-1.3.5/workflow/Snakefile", line 111, in __rule_preprocess
File "/media/zyshen/miniconda3/envs/snakemake/lib/python3.6/site-packages/snakemake/executors/__init__.py", line 568, in _callback
File "/media/zyshen/miniconda3/envs/snakemake/lib/python3.6/concurrent/futures/thread.py", line 56, in run
File "/media/zyshen/miniconda3/envs/snakemake/lib/python3.6/site-packages/snakemake/executors/__init__.py", line 554, in cached_or_run
File "/media/zyshen/miniconda3/envs/snakemake/lib/python3.6/site-packages/snakemake/executors/__init__.py", line 2352, in run_wrapper
Exiting because a job execution failed. Look above for error message
Removed: output/Preprocess/SortMeRNA/mt3namerep_interleaved.fastq
fastqc --outdir output/Preprocess/FastQC --threads 10 --extract output/Preprocess/SortMeRNA/mt3namerep_forward.fastq output/Preprocess/SortMeRNA/mt3namerep_reverse.fastq
Started analysis of mt3namerep_forward.fastq
Started analysis of mt3namerep_reverse.fastq
Approx 5% complete for mt3namerep_forward.fastq
Approx 5% complete for mt3namerep_reverse.fastq
Approx 10% complete for mt3namerep_forward.fastq
Approx 10% complete for mt3namerep_reverse.fastq
Approx 15% complete for mt3namerep_forward.fastq
Approx 15% complete for mt3namerep_reverse.fastq
Approx 20% complete for mt3namerep_forward.fastq
.......
zhiyong
https://gitlab.bio.di.uminho.pt/iquasere/MOSCA/-/issues/10File not found error2020-08-11T21:48:07ZJoão SequeiraFile not found error*Created by: MingL196*
Hi,
I tried running MOSCA using the command:
python MOSCA/scripts/mosca.py --files /hpc/home/mzl5860/Ming_workdir/G549/MOSCA/files_rename/G549-01-R1.fastq.gz,/hpc/home/mzl5860/Ming_workdir/G549/MOSCA/files_r...*Created by: MingL196*
Hi,
I tried running MOSCA using the command:
python MOSCA/scripts/mosca.py --files /hpc/home/mzl5860/Ming_workdir/G549/MOSCA/files_rename/G549-01-R1.fastq.gz,/hpc/home/mzl5860/Ming_workdir/G549/MOSCA/files_rename/G549-01-R2.fastq.gz --output output
The program ran up the FASTQC step, but then returned the error:
Traceback (most recent call last):
File "MOSCA/scripts/mosca.py", line 170, in <module>
reporter.info_from_preprocessing(args.output, mg_name, mg[0])
File "/ri/shared/modules7/miniconda3/envs/mosca-1.0.2/MOSCA/scripts/report.py", line 112, in info_from_preprocessing
self.info_from_fastqc(output_dir, name, '[Initial quality assessment]', prefix2terms)
File "/ri/shared/modules7/miniconda3/envs/mosca-1.0.2/MOSCA/scripts/report.py", line 76, in info_from_fastqc
prefix2terms[prefix][0], name, prefix2terms[prefix][i])) for i in [1, 2]]
File "/ri/shared/modules7/miniconda3/envs/mosca-1.0.2/MOSCA/scripts/report.py", line 76, in <listcomp>
prefix2terms[prefix][0], name, prefix2terms[prefix][i])) for i in [1, 2]]
File "/ri/shared/modules7/miniconda3/envs/mosca-1.0.2/MOSCA/scripts/mosca_tools.py", line 584, in parse_fastqc_report
file = open(filename).read().split('\n')
FileNotFoundError: [Errno 2] No such file or directory: 'output/Preprocess/FastQC/G549-01-R1.fastq.gz_R1_fastqc/fastqc_data.txt'
The program seems to be trying to read from the wrong file. Is there some format that the file names need to be in?
Can this be fixed?
Thanks.https://gitlab.bio.di.uminho.pt/iquasere/MOSCA/-/issues/12New problem encounter2021-03-01T17:08:38ZJoão SequeiraNew problem encounter*Created by: szypanther*
Hi iquasere,
Sorry to trouble you again, Here I still encounter another new problem after running almost 2 days later.
....
5500000 alignment record pairs processed.
5600000 alignment record pairs pro...*Created by: szypanther*
Hi iquasere,
Sorry to trouble you again, Here I still encounter another new problem after running almost 2 days later.
....
5500000 alignment record pairs processed.
5600000 alignment record pairs processed.
5700000 alignment record pairs processed.
5800000 alignment record pairs processed.
5834173 alignment pairs processed.
[Tue Feb 2 15:08:24 2021]
Finished job 9.
1 of 5 steps (20%) done
[Tue Feb 2 15:08:24 2021]
rule join_information:
input: output/Annotation/uniprotinfo.tsv, output/Annotation/Sample/aligned.blast, output/Annotation/Sample/reCOGnizer_results.xlsx, output/Metatranscriptomics/mtname.readcounts, output/Annotation/mgname.readcounts
output: output/MOSCA_Protein_Report.xlsx, output/MOSCA_Entry_Report.xlsx, output/Metatranscriptomics/expression_matrix.tsv
jobid: 5
threads: 12
Job counts:
count jobs
1 join_information
1
python /media/zyshen/work/MOSCA/MOSCA-1.2.2/workflow/join_information.py -e output/experiments.tsv -t 12 -o output -if tsv -nm TMM
2021-02-02 07:08:28: Joining data for sample: Sample
Traceback (most recent call last):
File "/media/zyshen/work/MOSCA/MOSCA-1.2.2/workflow/join_information.py", line 147, in <module>
Joiner().run()
File "/media/zyshen/work/MOSCA/MOSCA-1.2.2/workflow/join_information.py", line 58, in run
sheet_names = pd.ExcelFile(recognizer_filename).sheet_names
File "/media/zyshen/miniconda3/envs/mosca-1.2.2/lib/python3.6/site-packages/pandas/io/excel/_base.py", line 867, in __init__
self._reader = self._engines[engine](self._io)
File "/media/zyshen/miniconda3/envs/mosca-1.2.2/lib/python3.6/site-packages/pandas/io/excel/_xlrd.py", line 22, in __init__
super().__init__(filepath_or_buffer)
File "/media/zyshen/miniconda3/envs/mosca-1.2.2/lib/python3.6/site-packages/pandas/io/excel/_base.py", line 353, in __init__
self.book = self.load_workbook(filepath_or_buffer)
File "/media/zyshen/miniconda3/envs/mosca-1.2.2/lib/python3.6/site-packages/pandas/io/excel/_xlrd.py", line 37, in load_workbook
return open_workbook(filepath_or_buffer)
File "/media/zyshen/miniconda3/envs/mosca-1.2.2/lib/python3.6/site-packages/xlrd/__init__.py", line 170, in open_workbook
raise XLRDError(FILE_FORMAT_DESCRIPTIONS[file_format]+'; not supported')
xlrd.biffh.XLRDError: Excel xlsx file; not supported
[Tue Feb 2 15:08:28 2021]
Error in rule join_information:
jobid: 0
output: output/MOSCA_Protein_Report.xlsx, output/MOSCA_Entry_Report.xlsx, output/Metatranscriptomics/expression_matrix.tsv
RuleException:
CalledProcessError in line 232 of /media/zyshen/work/MOSCA/MOSCA-1.2.2/workflow/Snakefile:
Command 'set -euo pipefail; python /media/zyshen/work/MOSCA/MOSCA-1.2.2/workflow/join_information.py -e output/experiments.tsv -t 12 -o output -if tsv -nm TMM' returned non-zero exit status 1.
File "/media/zyshen/miniconda3/envs/mosca-1.2.2/lib/python3.6/site-packages/snakemake/executors/__init__.py", line 2340, in run_wrapper
File "/media/zyshen/work/MOSCA/MOSCA-1.2.2/workflow/Snakefile", line 232, in __rule_join_information
File "/media/zyshen/miniconda3/envs/mosca-1.2.2/lib/python3.6/site-packages/snakemake/executors/__init__.py", line 568, in _callback
File "/media/zyshen/miniconda3/envs/mosca-1.2.2/lib/python3.6/concurrent/futures/thread.py", line 56, in run
File "/media/zyshen/miniconda3/envs/mosca-1.2.2/lib/python3.6/site-packages/snakemake/executors/__init__.py", line 554, in cached_or_run
File "/media/zyshen/miniconda3/envs/mosca-1.2.2/lib/python3.6/site-packages/snakemake/executors/__init__.py", line 2352, in run_wrapper
Exiting because a job execution failed. Look above for error message
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /media/zyshen/work/MOSCA/MOSCA-1.2.2/.snakemake/log/2021-02-02T143849.506822.snakemake.log
**you can see the size of mtname_bowtie2_report.txt is zero.**
-rw-rw-r-- 1 zyshen zyshen 630 2月 2 08:27 mtname.log
-rw-rw-r-- 1 zyshen zyshen 13M 2月 2 08:36 mtname.readcounts
-rw-rw-r-- 1 zyshen zyshen 4.5G 2月 2 08:27 mtname.sam
-rw-rw-r-- 1 zyshen zyshen 0 2月 2 08:26 mtname_bowtie2_report.txt
any suggestion? Thanks!https://gitlab.bio.di.uminho.pt/iquasere/MOSCA/-/issues/2Empty Output Directories2019-05-23T09:59:39ZJoão SequeiraEmpty Output Directories*Created by: chandnisidhu*
Hi,
I have used my metagenomic data to check this pipeline. The pipeline worked well but the output directories generated were empty with no information. Please look into the matter.
nbhb-001-017:MOSC...*Created by: chandnisidhu*
Hi,
I have used my metagenomic data to check this pipeline. The pipeline worked well but the output directories generated were empty with no information. Please look into the matter.
nbhb-001-017:MOSCA-master csid$ python main.py --files /Users/csid/Documents/SRP187387_1.fastq /Users/csidhu/Documents/SRP187387_2 -o /Users/csid/Documents/
{ 'annotation_database': 'Databases/Annotation/uniprot.dmnd',
'assembler': 'metaspades',
'conditions': None,
'files': [ '/Users/csid/Documents/SRP187387_1.fastq',
'/Users/csid/Documents/SRP187387_2'],
'metaproteomic': False,
'no_annotation': False,
'no_assembly': False,
'no_binning': False,
'no_preprocessing': False,
'output': '/Users/csid/Documents',
'output_level': 'min',
'type_of_data': 'paired'}
{'files': ['/Users/csid/Documents/SRP187387_1.fastq', '/Users/csid/Documents/SRP187387_2'], 'type_of_data': 'paired', 'assembler': 'metaspades', 'annotation_database': 'Databases/Annotation/uniprot.dmnd', 'output': '/Users/csidhu/Documents', 'no_preprocessing': False, 'no_assembly': False, 'no_annotation': False, 'no_binning': False, 'output_level': 'min', 'metaproteomic': False, 'conditions': None}
Files
Experiment1
MG files: /Users/csid/Documents/SRP187387_1.fastq
Experiment2
MG files: /Users/csid/Documents/SRP187387_2
Type of data: paired
Assembler: metaspades
Annotation database: Databases/Annotation/uniprot.dmnd
Output: /Users/csid/Documents
No preprocessing: False
No assembly: False
No annotation: False
No binning: False
Output level: min
Metaproteomic: False
Conditions: None
Creating directories at /Users/csid/Documents
Created /Users/csid/Documents/Preprocess/FastQC
Created /Users/csid/Documents/Preprocess/Trimmomatic
Created /Users/csid/Documents/Preprocess/SortMeRNA
Created /Users/csid/Documents/Assembly
Created /Users/csid/Documents/Annotation
Created /Users/csid/Documents/Metatranscriptomics_analysis
Splitting reads in /Users/csid/Documents/SRP187387_1.fastq to /Users/csid/Documents/Preprocess/SRP187387_1_R1.fastq and /Users/csid/Documents/Preprocess/SRP187387_1_R2.fastq.
Handling SRP187387_1
Preprocessing of metagenomic reads has begun
Assembly has begun
Annotation has begun
Binning has begun
Splitting reads in /Users/csid/Documents/SRP187387_2 to /Users/csid/Documents/Preprocess/SRP187387_2_R1.fastq and /Users/csid/Documents/Preprocess/SRP187387_2_R2.fastq.
Handling SRP187387_2
Preprocessing of metagenomic reads has begun
Assembly has begun
Annotation has begun
Binning has begun
nbhb-001-017:MOSCA-master csid$https://gitlab.bio.di.uminho.pt/iquasere/MOSCA/-/issues/7self.input error in annotation step2019-04-24T14:02:35ZJoão Sequeiraself.input error in annotation step*Created by: bwu62*
When running mosca.py, an error is returned on line 147 since the Annotater( ) object never receives either an "input" argument or an "mg_name" argument, which both causes line 531 of annotation.py to crash since it ...*Created by: bwu62*
When running mosca.py, an error is returned on line 147 since the Annotater( ) object never receives either an "input" argument or an "mg_name" argument, which both causes line 531 of annotation.py to crash since it requires them.https://gitlab.bio.di.uminho.pt/iquasere/MOSCA/-/issues/6self.bowtie2 not defined in assembly.py2019-04-24T14:02:44ZJoão Sequeiraself.bowtie2 not defined in assembly.py*Created by: bwu62*
In assembly.py, line 113, self.bowtie2 throws an error because bowtie2 was never defined. Seems like an easy error to catch that should have been obvious if this pipeline was tested at all.*Created by: bwu62*
In assembly.py, line 113, self.bowtie2 throws an error because bowtie2 was never defined. Seems like an easy error to catch that should have been obvious if this pipeline was tested at all.https://gitlab.bio.di.uminho.pt/iquasere/MOSCA/-/issues/5typo in assembly.py2019-04-24T14:02:49ZJoão Sequeiratypo in assembly.py*Created by: bwu62*
In assembly.py, line 111, i think it should be "alignment.log" not "alignment.loj"?*Created by: bwu62*
In assembly.py, line 111, i think it should be "alignment.log" not "alignment.loj"?https://gitlab.bio.di.uminho.pt/iquasere/MOSCA/-/issues/4typo in mosca_tools.py?2019-04-24T14:02:55ZJoão Sequeiratypo in mosca_tools.py?*Created by: bwu62*
In the file mosca_tools.py, line 124 returns an error; if blast is None, then blast.replace( ) cannot run since None has no replace method. Should it be reference.replace( ) instead? The next line seems to imply the ...*Created by: bwu62*
In the file mosca_tools.py, line 124 returns an error; if blast is None, then blast.replace( ) cannot run since None has no replace method. Should it be reference.replace( ) instead? The next line seems to imply the check should be for reference object not blast.https://gitlab.bio.di.uminho.pt/iquasere/MOSCA/-/issues/1Several suggested improvements to install.bash2019-04-24T14:01:58ZJoão SequeiraSeveral suggested improvements to install.bash*Created by: bwu62*
Things I think you should consider changing:
1. Remove Xming install line at beginning of file. This seems to be specifically a Windows program and should not be hard coded into the automated install file.
1. As ...*Created by: bwu62*
Things I think you should consider changing:
1. Remove Xming install line at beginning of file. This seems to be specifically a Windows program and should not be hard coded into the automated install file.
1. As of writing this, installing conda and then trying to conda install anything runs into SSL errors due to a change in pyOpenSSL. You need to upgrade it to at least 16.2.0 with `python3 -m easy_install --upgrade pyOpenSSL==16.2.0`
1. Conda installs should have an additional -y flag to suppress confirmation
1. Lines 23,31 you shouldn't hard code your own home directory path.
1. Line 25,31 please prepend any comments with a #
1. Line 35 I don't think you need to install aptitude just to install build-essential. Why not add build-essential to line 36 and do both at once?
1. Lines 73,74 are python code and do not belong in a bash filehttps://gitlab.bio.di.uminho.pt/iquasere/MOSCA/-/issues/3Possible to run MT without MG?2021-02-01T13:00:00ZJoão SequeiraPossible to run MT without MG?*Created by: bwu62*
Hi,
Thanks for the recent overhaul to the software. I was wondering if it is possible to run MOSCA only on MT data without MG data? I have some MT data simulated with Polyester and art_illumna (which only simulate...*Created by: bwu62*
Hi,
Thanks for the recent overhaul to the software. I was wondering if it is possible to run MOSCA only on MT data without MG data? I have some MT data simulated with Polyester and art_illumna (which only simulate RNA-seq data) with no associated MG reads and I was wondering if it's possible to do DE with it.
Thanks.