Genomics Data Pipeline
Last reviewed
IntermediateWhat you'll learn
~50 min- Ask an AI tool, in plan mode, for a program that reads the summaries a read-quality tool wrote for each read file of a project (totals and module results marked pass, warn or fail) and puts them in one table and a short report
- Give it your facility's rules, and read its plan for what matters: both read files of a sample lined up, results judged against what your facility expects for that kind of library, a reason beside every flag, and a log of every file read with its size and checksum
- Check the report against values worked out for the practice files, and check that two runs on the same files give the same report and log
- Say why the same fail is expected for one kind of library and a reason to look for another, and who decides what happens to a sample
The story, and what you will end up with
It is Tuesday 10 November 2026, at a genomics core. You are the core’s bioinformatician: for every project a lab sends through the core you look at read quality, and before analysis starts you tell the lab which samples you want to talk about. This morning it is project P-2611 from the lab LAB-42: nine samples, sequenced paired-end (explained in the box below), six RNA-seq libraries and three whole-genome libraries made without PCR.
Before this morning you did what comes first. The sequencing staff checked the run itself. You read the demultiplexing report, the summary written when the reads were sorted to samples, for each sample’s share of the reads. You checked the read files’ checksums against the delivery, and that the read length matches what was ordered. You compared the core’s sample sheet for the project with the lab’s submission form: each sample’s library kind, and how many read pairs it asked for. You have the lab’s own measurements from submission at hand, such as concentration, fragment size and RNA integrity. Then the core’s pipeline ran its standard read-quality tool on every read file, at your facility’s recorded version and settings, and saved a small summary for each file.
That is eighteen summaries, read one at a time. What you want is one table for the project: every sample, both its read files, each module’s result and the read count against the request, with the samples that need your look set apart and the reason beside each. And because you may be asked months from now what this table was made from, you want a log of exactly which files it read, with each file’s size and checksum, so the same table can be made again and shown to be the same.
By the end of this lesson you will have a small program on your own computer that does that. It calculates no quality figure of its own: the read-quality tool did that work. It reads the results the tool wrote, lines them up by sample, compares them with the sample sheet and your facility’s list of the results it expects for each kind of library, works out each sample’s share of its request, and lists the samples that need your look, with the reason for each. You build it by asking your AI tool in plain English, and you check it against numbers this lesson gives you.
🔍Words used in this lessonclick to expand
- Paired-end: each DNA fragment is read from both ends. Each sample has two read files, R1 and R2, with one read in each for every pair, so the two files hold the same number of reads.
- Read-quality tool: software that reads a read file and writes a summary of it. Its results come in parts called modules, such as per base sequence quality or adapter content, each marked pass, warn or fail by the tool’s own limits.
- Library kind: how the sequenced material was prepared, such as RNA-seq (from RNA) or whole genome (from DNA). The sample sheet records it for each sample.
- Expected list: your facility’s own list of the module results it expects to see warn or fail for each library kind, so they are not a reason to look on their own.
- Checksum: a fingerprint of a file’s exact contents. SHA-256 is one way to make it. If a single character in the file changes, the checksum changes.
Before you start
First, if your AI tool is still open, leave it: type /exit and press Enter (if you use Antigravity, press Ctrl+D with nothing typed instead), and wait until you see the $ or % prompt. The next line goes there, in the terminal, not in the AI tool. It makes a folder for this lesson beside your practice folder and goes into it:
mkdir -p ~/bio-workbench/03-qc-table && cd ~/bio-workbench/03-qc-tableStart your AI tool with the start line you kept from the setup lesson, the one with approval prompts off. Then ask it to set the folder up:
Set this folder up so we can undo any change you make, download https://learn.intellimetrics.net/starters/biotech/03/qc-summaries.zip and unpack it into a data folder, and save a checkpoint.Look for it saying three things: the folder is set up with Git (the program that saves snapshots, from setup), the data folder holds samples.csv, expected-by-library-kind.csv and a summaries folder of 18 files, and a first snapshot, your checkpoint, is saved. Later you can ask whether the files have changed since then. If it does not mention one of them, ask: “Did you save a checkpoint with the files in it?” It may also keep the zip file in the data folder, which is fine; if it says it kept one somewhere outside this folder, ask it to delete that copy. If the download does not work, download qc-summaries.zip in your browser, check where it went (the browser’s list of downloads shows the folder), and tell your AI tool, for example: “qc-summaries.zip is in my Downloads folder. Unpack that one file into a data folder here, and save a checkpoint.” In Windows PowerShell or a desktop app, “my Downloads folder” works as is; only say “my Windows Downloads folder” if you use Ubuntu (WSL) on Windows, which keeps its own Downloads folder apart from Windows.
The box below says what is in the practice files; later steps refer to them.
ℹWhat is in qc-summaries.zipclick to expand
Twenty small files, each of which opens in a spreadsheet program or a text editor.
samples.csv, the core’s sample sheet for project P-2611: Sample_ID, Lab, Project, Library_kind, Read_length, Read_pairs_requested.
expected-by-library-kind.csv, this practice facility’s own expected list: for RNA-seq, a warn or fail is expected on per base sequence content and on sequence duplication levels; for its whole-genome libraries made without PCR, none.
summaries/, one file per read file, named like S-01_R1_summary.txt. Each holds the sample, the read (R1 or R2), the read file’s name, its total reads, read length and GC percent, then eleven module results, each pass, warn or fail.
| Sample | Library kind | Requested read pairs | Total reads R1 | Total reads R2 | Results that are not pass (both reads unless marked) |
|---|---|---|---|---|---|
| S-01 | RNA-seq | 20,000,000 | 18,204,990 | 18,204,990 | Per base sequence content fail; sequence duplication levels warn |
| S-02 | RNA-seq | 20,000,000 | 19,880,412 | 19,880,412 | Per base sequence content fail; sequence duplication levels fail |
| S-03 | RNA-seq | 20,000,000 | 21,450,007 | 21,450,007 | Per base sequence content fail; sequence duplication levels fail; adapter content fail (R1), warn (R2) |
| S-04 | RNA-seq | 20,000,000 | 14,880,221 | 14,880,221 | Per base sequence content fail; sequence duplication levels warn |
| S-05 | RNA-seq | 20,000,000 | 20,912,334 | 15,406,118 | Per base sequence content fail; sequence duplication levels fail |
| S-06 | RNA-seq | 20,000,000 | 22,617,503 | 22,617,503 | Per base sequence content fail; sequence duplication levels fail; per sequence GC content warn |
| S-07 | Whole genome PCR-free | 30,000,000 | 31,775,640 | 31,775,640 | Sequence duplication levels fail |
| S-08 | Whole genome PCR-free | 30,000,000 | 33,018,256 | 33,018,256 | None |
| S-09 | Whole genome PCR-free | 30,000,000 | 29,640,118 | 29,640,118 | None |
The RNA-seq libraries were read at 101 bases, the whole-genome libraries at 151. GC percent is 47 to 49 for every RNA-seq sample except S-06, at 53; 41 to 42 for the whole-genome samples. The read counts are reads that passed the instrument’s filter.
Everything in these files is made up for this course: every sample, lab, project and number, and this practice facility’s expected list. In the story the core’s pipeline wrote the summaries; like everything else here they are made up, and no real sample or organism is behind them. Each lesson in this track has its own practice files, with their own people and dates.
Switch to plan mode
Your AI tool is already open. Switch to plan mode now: Shift+Tab until it says plan, or /plan (the keys for each tool). In plan mode it usually reads your files, asks its questions and shows you a plan before it builds anything; the order can vary.
The conversation
Each message below is an example. Say it your own way.
Say what you want
Say what you do, which files to use, and what you want from them:
I check read quality for projects at our genomics core. Build a program that puts each project's quality summaries, like those in data, into one table and report, logging exactly which files it read. Ask me what you need first.Your AI tool usually reads the files and comes back with questions. It may already point out some of what it sees: S-05’s two read files with different totals, S-04 short of its request, S-03’s adapter content, S-06’s GC, S-07’s duplication.
Give it your rules
These are this practice facility’s own rules, numbers included. Send them as they are, so your report can be checked against the figures on this page; your own facility’s rules come in under Your own projects, near the end.
Our rules: a sample needs a look if any read file has a warn or fail not on our expected list for its library kind, or under 90% of requested read pairs.Wait for its reply, then send the rest in a second message:
It also needs a look if its two read files differ in total reads, or a summary is missing. Say fine or needs a look, never passed. Log each file's SHA-256 checksum and size.Answer its questions
Answer each in a sentence. These are questions the tools often ask, with one way to answer each. Yours may ask them in a different order or show a list of choices:
| If it asks | You might say |
|---|---|
| One row per sample, or one per read file? | One row per sample, with both read files’ results. |
| Will each project come as a folder like this one? | Yes, each project is its own folder like this one. |
| Read pairs from R1, or the smaller of R1 and R2? | Use the smaller count. |
| A missing or unreadable summary: stop, or note it and carry on? | Note it, mark the sample needs a look, and carry on. |
| Should the log also list files it skipped? | Yes, with the reason each was skipped. |
| What should the outputs be? | A table my spreadsheet program opens, a short report, and the log. |
| A program in the terminal, or a page in the browser? | A program I run in the terminal. |
| Should it also read other summary layouts? | Only this layout for now. |
If it asks several questions at once, answer the ones about which samples need a look and how reads are counted from the table above and the rules you sent, a sentence each, and accept its suggestions only about how the table and report look. If you cannot tell which kind a question is, treat it as a rule. If it asks something not listed here, answer it the same way, in a sentence.
Read the plan
Next it usually shows you a plan: what it will build and how it will check it. Read it for six things:
- Each sample’s R1 and R2 summaries are lined up by sample, and a missing summary, or a summary for a sample not on the sheet, makes the sample need a look.
- Results are judged against your expected list for the sample’s library kind: a warn or fail on the list does not put the sample under needs a look, one that is not on the list does, and the table still shows every module’s result for both reads.
- Read pairs are compared with the request, and the two read files’ totals with each other.
- Every sample that needs a look gives its reason, with the module and read or the numbers, and the report says fine or needs a look, never passed, failed or ready for analysis.
- The log lists every file read, with its name, size in bytes and SHA-256 checksum, and the date it ran; it holds no time of day, so two runs on the same files on the same day give the same report and the same log.
- It is a program you can run again on the next project’s folder, and your files are read and left as they are.
A first plan often leaves one of these out, most often item 5: it may write the time each file was read. Say so in a sentence, in your own words, for example:
Log the date it ran, not the time, so two runs on the same files on the same day give exactly the same report and log.When the plan has all six, approve it: if your AI tool offers a choice to go ahead, pick it; otherwise say “Yes, go ahead and build it.” In Codex, press Shift+Tab first to leave plan mode. In Copilot, saying it brings up a choice to go ahead; pick that. Let it build. It may run what it built and tell you how to run it yourself; either way, the next step asks it to. When it has finished, ask it: “Did you install anything, and where?” Nothing, or something inside this lesson’s folder, is fine for these lessons. If it names anywhere else on your computer, such as the zip file it downloaded, ask it what that is for, and to remove it if you do not want it there.
Look at what it built
Ask it to run what it built:
Run it on the files in data and show me the report.Your report will be worded and laid out in its own way, and some figures may be only in the table file, not in the report. Check it for these:
- Five samples need a look. S-03: adapter content fail on R1 and warn on R2. S-04: 14,880,221 of 20,000,000 read pairs, 74.4%. S-05: R1 has 20,912,334 reads and R2 15,406,118. S-06: per sequence GC content warn on both reads. S-07: sequence duplication levels fail on both reads, which the practice expected list does not expect for a whole-genome library.
- Four samples are fine: S-01 (18,204,990 of 20,000,000, 91.0%), S-02, S-08 and S-09. S-01’s and S-02’s per base sequence content and duplication results are on the expected list for RNA-seq, and the table still shows them. S-08 and S-09 have no result other than pass.
- S-05 shows 77.0% of its request if the program counts read pairs from the smaller of its two files, as the answer above suggests; it needs a look either way.
- The log lists 20 files: the two sheets and 18 summaries.
samples.csvis 470 bytes with a SHA-256 checksum beginning852433c0,expected-by-library-kind.csv125 bytes beginning5d5bb962, and each summary 418 bytes;S-07_R1_summary.txtbegins27a7f50e. - No line calls a sample passed, failed or ready for analysis. The module results themselves may say pass, warn or fail, as the quality tool wrote them.
- When you ask “Run it again on the same files. Are the report and log exactly the same as the first time, and have my files changed since the checkpoint?”, it says the report and the log are the same, and that your files have not changed.
If something on the list is not true, tell your AI tool in a sentence what you see and what you expected, for example: “S-07 shows as fine. Its library kind is whole genome, and nothing is expected to fail for that.”
Notice what one word on the sheet decides
Put two rows of your report side by side: S-02 and S-07. Both have sequence duplication levels fail on both reads. S-02 is fine and S-07 needs a look. Their duplication results are the same. What separates them is the sheet: S-02 is RNA-seq and S-07 is whole genome.
Now see how much rests on that one word. Ask your AI tool:
Copy the data folder into this folder. In the copy only, change S-07's library kind to RNA-seq. Run the program on the copy and show me what changed in the report and the log.Before you read on: what do you expect for S-07, and how many lines of the log will change?
S-07 moves to fine; its reads and their results are the same as before. In the log one line changes, the sample sheet’s: samples.csv is now 456 bytes, 14 fewer if only those words changed, with a new checksum, and the expected list and all 18 summaries keep theirs. So the log shows which input changed, and the report shows what it changed. Neither can tell you which version of the sheet is right. That is why you compare the sheet’s library kinds with the lab’s submission form before you trust the table at all, and look again when a sample’s figures do not fit its kind (S-07’s 151-base reads and 41% GC sit with S-08 and S-09, not with the RNA-seq samples).
Whether S-07’s duplication is a problem is not in these files either. The documentation of one widely used read-quality tool, FastQC, says what a warning or fail in its duplication module means: “A warning or error in this module is simply a statement that you have exhausted the diversity in at least part of your library and are re-sequencing the same sequences” (FastQC help, Duplicate Sequences, read 11 November 2026). The same page says a high level of duplication “is more likely to indicate some kind of enrichment bias (eg PCR over amplification)”. For a whole-genome library made without PCR, a person asks what caused it, including something innocent. The same page names two kinds of duplicate, “technical duplicates arising from PCR artefacts, or biological duplicates which are natural collisions where different copies of exactly the same sequence are randomly selected”, and says “From a sequence level there is no way to distinguish between these two types”. In this practice core the lab’s submission measurements and the library’s record are where you look first; check your own core’s procedures.
The same holds for S-03. For its adapter content module the same documentation says: “Any library where a reasonable proportion of the insert sizes are shorter than the read length will trigger this module. This doesn’t indicate a problem as such - just that the sequences will need to be adapter trimmed before proceeding with any downstream analysis” (Adapter Content, read 11 November 2026). In this practice core you also look at the RNA integrity and fragment size from submission before calling it a trimming matter; your own core’s procedures say what it checks.
S-05 and S-04 are different. The two read files of a pair hold the same number of reads, and S-05’s do not. In this practice core that is the core’s own matter: you fetch or sort the files again, and run the quality tool on them again, before anything is said to the lab. S-04’s shortfall goes first to the demultiplexing report, to see whether its share of the pooled reads was low. Check your own core’s procedures for what comes first in each case.
When you have looked, remove the copy: “Delete that copy and what you made from it; I’m done with them.”
The table says which results differ from what your facility expects for that kind of library; whether a sample goes on to analysis, is sequenced again or is discussed with the lab is the bioinformatician’s and the lab’s decision, under the core’s procedures and its service agreement.
A sample on the fine side has met your rules, and that is all. It is not ready for analysis until you have done everything else your facility requires.
One real run
Here is part of what one AI tool said after the experiment above, with Claude Code on 11 November 2026. Yours will word it differently; check your report against the checks above, not against this.
The program accepts the library kind from the sample sheet as givenand doesn't cross-check it. S-07's data looks like a whole-genomelibrary rather than RNA-seq: 151-base reads, 30M pairs requested,41% GC, the same as S-08 and S-09. So a wrong library kind in thesheet can hide a real problem, and the checksum in the log is theonly sign the sheet changed.The last sentence says more than the files can. A changed checksum shows only beside an earlier log of the same project, so keep each log with the table it belongs to. And a changed checksum says the sheet changed, not which version is right; the submission form says that.
The samples it talks about are made up for this course.
How real projects differ
- The tool’s results are pointers. The maker of FastQC writes: “It is important to stress that although the analysis results appear to give a pass/fail result, these evaluations must be taken in the context of what you expect from your library”, and “You should treat the summary evaluations therefore as pointers to where you should concentrate your attention and understand why your library may not look random and diverse” (FastQC help, Evaluating Results, read 11 November 2026).
- Your expected list is your facility’s. The practice list has two kinds of library. The same documentation says of per base sequence content: “Nearly all RNA-Seq libraries will fail this module because of this bias, but this is not a problem which can be fixed by processing, and it doesn’t seem to adversely affect the ablity to measure expression” (Per Base Sequence Content, read 11 November 2026, spelling as on the page). Check whether your facility’s list goes by more than RNA or DNA, such as how the library was made, whether with PCR, the organism, how deeply it was read and the instrument, and check who keeps it.
- GC is read against the organism. For per sequence GC content the documentation says: “An unusually shaped distribution could indicate a contaminated library or some other kinds of biased subset” (Per Sequence GC Content, read 11 November 2026). S-06’s warn is a reason to look at the plot, the over-represented sequences and any contamination screen your facility runs, not a finding of contamination.
- The request and the count must be in the same unit. Check whether your core’s requests are counted in reads or in pairs, or as depth of coverage for whole-genome work, and that your sample sheet’s request and the summaries’ counts are in the same unit before a rule like the practice 90% means anything. Check in your core’s service agreement whether a shortfall is owed more sequencing, who pays, and whether output is counted per sample, per lane or per pool.
- Check what your own files hold, such as several read files per sample, one per lane; index and UMI reads; samples topped up from a second run; file names that do not match the sample sheet exactly; or summaries that keep totals and module results in separate files.
- The log covers the table, not the reads. The practice log shows which summaries and sheets went into the table. Nothing in the practice summaries ties one to the exact read file it was made from beyond the file’s name. Check whether your facility’s record of a project keeps the read files’ own checksums, which read file each summary came from, the quality tool’s version and its settings, and where.
- Check what your core already runs. Ask whether a program already gathers your quality reports into one table, and check what it shows and logs before you build your own; this lesson’s program is small enough to read, and what it adds is your expected list, the request and the log.
- Check your files for sequence and identifiers. Check whether your full quality reports list over-represented sequences, which are read sequence, and whether your sample names carry a patient or animal identifier, and follow your facility’s rules for them as you would for the reads.
Keep going
Small changes do not need plan mode. Say each in a sentence, then run it again:
- “Draft a short note to LAB-42 about the samples we want to discuss, saying only what the files show, with a blank for what happens next.” Check every number in it, take out anything that is the core’s own matter first, such as S-05’s files, and fill in the blank only after you and the lab have decided; nothing goes to the lab until then.
- “Does the program name any sample or project from these practice files? If it does, make it work those out from the files instead.” A program built in a conversation may keep your examples written into it, and on the next project’s files it would get them wrong.
- “Save a checkpoint.” Your AI tool saves a snapshot you can return to.
Your own projects
When you want a table for a real project, use the finished program. It runs on your real files by itself, with your AI tool closed, so your AI tool never sees them. Nothing confidential or unpublished, and nothing about people or about animals under a protocol, goes to your AI tool or into this course folder; that includes data covered by an agreement, and sample names that identify people or animals. Your quality tool’s summaries will not be laid out like the practice files, and your sample sheet and expected list may live in a form or a document. Do these steps in order; the first five happen now, while your AI tool is open and this folder still holds only the practice files:
-
Go through the data check, the questions in setup about what the data is, who decides how it is used, and which AI tool would see it. A lab’s data is that lab’s to decide on, and human-sample data may need your facility director or compliance office as well. Go on only if they allow this use.
-
Decide on a folder of its own for your real files, outside this course folder, for example
~/qc-tables. Nothing has to be in it yet. If the folder is not there, ask your AI tool to make it: “Make an empty folder called qc-tables in my home folder.” -
Ask your AI tool for blank sheets and a made-up project to try: “Make me a blank sample sheet and a blank expected list my spreadsheet program can open, in the qc-tables folder in my home folder, and a made-up project here with one sample, and show the program working on it.” Look for it saying where it saved the two blank sheets and showing the made-up sample as fine or needing a look, with a reason. Then open one of your quality tool’s summaries for a practice or old project you may share, and tell your AI tool how it is laid out (if your tool writes totals and module results in separate files, say so and describe each file’s layout), with two summaries you make up in the same shape, made-up sample names and numbers, never a real one. For example: “Our tool writes one summary per read file, with a line for each module giving its result and name. Here are two made-up ones in the same shape. Change the program to read files like that.” Then paste the two you made up. Share only the layout, with made-up names and values; if your facility treats column names or the layout itself as confidential, ask whoever is responsible first, or make up equivalents. Whoever produces the summaries at your core, or its bioinformatician, can make up two in the right shape for you.
-
Give the program your facility’s own rules and expected list in place of the practice ones, from its written procedures, numbers included. For example: “Replace the practice rules and expected list with ours, which I’ll send next. Then make up one sample for each rule that only that rule flags, and show me the report.” Wait for its reply, then send your rules and your list. Check that each made-up sample needs a look for the rule it was made for, with that rule’s reason. Your AI tool may also replace the practice list in the data folder; that is fine now that you have checked the practice files.
-
Ask your AI tool for the exact line, for example: “Give me the exact one line to run this myself on a project folder inside a folder called qc-tables in my home folder, with full paths, saving the output there too.” Use your own folder’s name. If it offers to copy your real files into this folder instead, say no. Ask it too what the project folder must hold and what each file must be called. Copy the line it gives you into a note. It will look something like this one, though yours may differ:
Terminal window python3 /home/dana/bio-workbench/03-qc-table/qc_report.py /home/dana/qc-tables/PROJECT_NAME /home/dana/qc-tables/qc-outputRead it left to right.
python3runs a Python program. Next comes the program, in this lesson’s folder; then the project’s folder inside qc-tables, which holds its sample sheet, your expected list and its summaries; last, the folder the table, report and log are written to.PROJECT_NAMEstands for your project folder’s name; each project, you change it to that project’s./home/danais the home folder of someone whose user name is dana; your own user name goes wheredanais, and on a Mac the folders begin/Users/instead of/home/. The line runs the program where it is, so leave this lesson’s folder in place. If you are unsure about any part, ask your AI tool now, while it is still open. -
Leave your AI tool the way you did at the start of this lesson.
-
Put a real project’s summaries, its sample sheet filled in from the submission forms, and your expected list in a folder of their own inside qc-tables, named and laid out as your AI tool told you. Paste the line from your note at the
$or%prompt and press Enter. The table, report and log appear beside your real files, not in the course folder. To come back to the lesson folder later, type this lesson’s folder line again, the line near the top of this page that begins withmkdir. -
Before you act on the report, compare a few of its results with your quality tool’s own reports for the same files, and keep the log with the table. If they differ, find out why first.
Try another case
Before the quiz, write two or three sentences for yourself. Next month a project’s table says a small-RNA library needs a look: sequence duplication levels fail on both reads and per base sequence content fail, neither on your expected list, which has no row for small RNA. A colleague says to add both to the list and rerun. What do you check, and what do you decide?
A good answer finds out first whether this library kind was ever written into the facility’s list, and from what: the documentation the list rests on and the facility’s own experience with such libraries, not one project’s results. It talks to whoever keeps the list before it changes, and it reads this project’s samples against what that library kind should show before saying anything to the lab. Changing the list to make a flag go away would hide the next real problem; deciding what the list says is the facility’s job, under its procedures.
Key takeaways
- A short ask, then your rules. Say what you do and which files; give your facility’s rules and expected list in its own words.
- Read the plan for how files are matched and judged: R1 and R2 lined up, results read against the library kind, a reason beside every flag.
- The log is checked, not assumed: two runs on the same files on the same day give the same report and log, and a changed file shows as a changed checksum.
- A flag says where to look. What it means, and what happens to the sample, is the bioinformatician’s and the lab’s to decide.
- Practice data stays practice data: real projects go to the finished program, with your AI tool closed, after the data check.
An RNA-seq sample's two read files both show per base sequence content fail and sequence duplication levels warn. Nothing else is warn or fail, and it has 96% of its requested read pairs. Under this practice facility's rules and expected list, what does the report say?
A whole-genome sample needs a look for sequence duplication levels fail on both reads. A colleague suggests changing its library kind on the sample sheet to RNA-seq, since RNA-seq expects that result. What should happen?
What’s next
The next lesson is about looking at one exported mass spectrum and the peaks the instrument’s software called.