Data retention and who holds the raw files

The download link expired on a Friday, and the variant table was the only file anyone had copied onto a laptop. Retention is the decision about who still holds the reads after that Friday. It is a troubleshooting problem because projects discover it when something is already missing. The way those reads were produced is next-generation sequencing from library to reads. The place the decision should have been written is what to put in a sequencing statement of work, and the wider commissioning habit is commissioning a sequencing or proteomics study.

Three file classes are three different custodies

FASTQ holds base calls and quality scores. For paired-end runs it is usually two files per sample, and across a project it is the bulky class. It is also the class from which you can still trim again, align again, and call again. Losing it means the experiment survives only as other people's conclusions.

BAM holds alignments to a reference. CRAM holds a compressed form of that alignment and generally needs the same reference present before you can expand it. Either file is large. Either file is useless as a long-term object if you did not record the genome build. A BAM you cannot match to a build is a well-compressed ambiguity.

VCF holds variant calls: a table. It is small beside the reads. It is the worst sole archive, because every filter and every caller version is already baked in. A PDF or a spreadsheet of gene symbols is smaller still and is not a sequencing file at all. Sanger data are a different shape, often a trace and a base call for one reaction, and they are easy to lose for the opposite reason: they feel too small to inventory. Inventory them anyway.

This page stays with bulk and small, and it does not invent a file size. Those two ideas are what you need in order to choose a custodian. The exact footprint depends on depth, length, compression and how many samples you truly ran.

Who holds them, and for how long

Custodian means the party your institution will ask when a file is missing, and the party allowed to delete. A sequencing provider may hold a working copy. That does not make them the archive of record unless the agreement says so. A core facility on your campus may be the archive. Your own group may be. Split custody is legitimate if it is written: the provider keeps FASTQ for a stated period, you mirror it within that period, and you alone keep the VCF versions you filtered later.

Duration comes from institutional policy, funder expectations and the scientific need to reanalyse. This article will not invent a retention period. Write a number of years, or write "until the grant's archive rule", and name the rule. "As long as possible" ends on the day a disk is reused.

Public archives, including the Sequence Read Archive and the European Nucleotide Archive, are a third holder, available only when consent and policy allow deposit. A plan to deposit is not a backup you have completed. Deposit after you know the files are the ones you mean, with the metadata that makes them findable.

Encryption in transit, and a window that shuts

Moving sequence data is part of retention. A link in an email is a door. Prefer a transfer that encrypts the channel, and prefer a manifest of checksums so you know the bytes that arrived are the bytes that were sent. Encryption in transit does not encrypt the disk you then leave unlocked. It also does not decide who may download. Access control is a separate line in the agreement.

Providers close download windows. The closure is not a malfunction. If your copy is incomplete on that day, the troubleshooting starts from a disadvantage. Before the date, copy every class you are owed, verify checksums, and open at least one FASTQ and one alignment far enough to see headers. A checksum can match a file that is the wrong sample if you copied the wrong row of the manifest. Look at the name inside the file.

When a link has already died, ask whether a second copy still exists and under what conditions it can be regenerated from the instrument's own output. Sometimes the instrument output is already gone and the VCF is all that remains. Then the troubleshooting result is a limitation, and the paper should say so. Do not reconstruct raw reads from a variant table. You cannot.

A workflow that catches a missing file while you can still ask

On delivery day, check the sample count against the statement of work. Check that paired reads both exist. Check that the reference build is written in the same folder as the BAM. Compute checksums and store them somewhere other than the folder you might lose. If any item fails, tell the provider before the window moves. If the count is short by one sample, assume a naming error first and a failed library second. Both are easier to solve while the other side still has the disk.

If you discover the gap months later, write down what you still hold and stop analysing as if the missing sample were merely delayed. A silent missing control changes every comparison.

Failure looks like a tidy summary

The common failure is aesthetic. The summary table is easy to email, so it becomes the archive. The FASTQ lived on a link nobody opened. The BAM lived on a student laptop that was reimaged. Months later a reviewer asks for the reads supporting one genotype. You have the genotype and a story. That is a troubleshooting endpoint: the claim's evidence is gone. The repair, where any exists, is a provider who still holds the files. The prevention is the manifest on day one.

A related failure is a CRAM copied without the reference, then the reference URL drifts. Store the build's identity in your own records. Check it while the alignment still opens.

Policy, people, and sequence that identifies them

Human sequence can identify people and families. Who may hold it, who may download it, and how long it may be kept are institutional and legal questions. This article is not legal advice and not an ethics approval. It is a reminder to put the institution's answer into the sequencing agreement rather than inventing a casual mirror on a personal drive. Microbial and animal projects have their own sharing rules. Biobanking language in ISO 20387 shows how custody is discussed for biological resources. Sequence files deserve the same explicitness.

Encryption, access lists and deletion dates are controls. They do not make a research VCF into a clinical record. Do not store files as if they were one.

Unstable links and a drive handed across a city

Transfers inside India often meet the same operational facts: a link that stalls, a campus connection that drops in a storm, a power cut in the middle of a multi-hour copy. The risky repair is a portable drive passed by hand or by courier with no manifest and no named sender. The drive may be the right practical tool. It is also an unlogged copy unless you write who created it, which checksums it carries, and who received it. Verify those checksums before the sender deletes their side. Heat and a loose bag do not improve a disk. Do not treat a successful ping of a download page as evidence that the FASTQ finished. Look at the file size against the manifest, and look at the checksum.

What the retention lines of an enquiry should say

Name the custodian for FASTQ, for alignments, and for VCF. Name the duration and the date a download window closes. Name encryption for the transfer and the checksum method. Say whether Sanger traces are in the same archive. The scientific context is the genomics research pathway. Reagents are a separate catalogue question in the genomics and sequencing catalogue, and they do not store your files.

A genome project can be discussed through the whole-genome sequencing enquiry reference. A trace project can be discussed through the Sanger DNA sequencing enquiry reference. Put the custody lines in the quote request. Agree them before the instrument runs. After the link has closed, the enquiry is a search, not a design.