LIFTOVER_ANNOTATE¶
The LIFTOVER_ANNOTATE command looks up where each input row lands in another genome build and appends the result as extra columns. Unlike LIFTOVER it does not move the rows: every input row produces exactly one output row, in the same place and in the same order, with its original coordinates, followed by a liftover status and the lifted-over location. No row is dropped, merged or reordered.
Because the rows stay in place the output is still sorted by the input coordinates, no sort genome is needed and
the command is safe to use under pgor. The lifted-over coordinates are ordinary columns, so you can filter on the
status, keep only the mapped rows, or relocate the rows yourself (see the notes below).
The mapping between the builds is read from a chain file, a GOR file with the header
Chrom tStart tEnd qChrom qStart qEnd qStrand qScore (see Chain format), typically generated from a UCSC chain
file. The chain can also be a nested query, <(gor ...).
The command must know the type of the data, and exactly one of -snp, -var, -seg and -seg1 is required.
Usage¶
gor ... | LIFTOVER_ANNOTATE <chain> ( -snp | -var | -seg | -seg1 ) [<options>]
nor ... | LIFTOVER_ANNOTATE <chain> ( -snp | -var ) -chrom <col> -pos <col> [<options>]
nor ... | LIFTOVER_ANNOTATE <chain> ( -seg | -seg1 ) -chrom <col> -pos <col> -end <col> [<options>]
<chain> is a chain file or a nested query <(gor ...).
Options¶
Input type (exactly one is required):
|
A single position. |
|
A variant. The position is |
|
A segment in GOR coordinates, half-open: |
|
A segment in 1-based inclusive coordinates: |
Input columns:
|
The chromosome column. Default is column 1 ( |
|
The position column (the start for |
|
The end column. Only valid with |
|
The reference allele column. Only valid with |
|
The alternate allele column. Only valid with |
-var requires both a REF and an ALT column. Columns can be given by name or by number. Column names are matched
case-insensitively.
Output and behaviour:
|
Do not fill equal gaps between the blocks of a chain (see Equal gaps): a position in
such a gap is |
|
Prefix of the appended column names. Default is |
|
Comma separated list of source contigs that are copied through unchanged, with the
status |
|
Comma separated list of the contigs that are accepted, both as source contigs and as
target contigs. A row on another source contig, or one that lifts to another contig,
gets the status |
|
What to do when blocks of the chain overlap on the source side. |
|
Comma separated list of statuses. The query fails on the first row that gets one of them. The statuses are checked against the statuses of the input type (see Status values) when the query is parsed. |
|
Only valid with |
|
Only valid with |
|
Only valid with |
If an appended column has the same name as a column of the input the query fails with an error. Use -prefix to
choose other names.
Output columns¶
The columns below are appended to the input columns, in this order. The names are shown with the default prefix
liftover_. A row with a status that has no target location (for example unmapped) gets an empty location: the
empty string in the target contig and strand columns (and in liftover_ref and liftover_alt for -var) and
0 in the position columns (liftover_pos, or liftover_start and liftover_stop). The other columns
(liftover_status, and liftover_chain_blocks and the -diag columns of -seg and -seg1) always have
values.
-snp: liftover_status, liftover_chrom, liftover_pos (I, the 1-based target position),
liftover_strand (S).
-var: liftover_status, liftover_chrom, liftover_pos (I, the 1-based target position of the start of
the variant), liftover_ref (S), liftover_alt (S), liftover_strand (S). When the target strand is -
each allele is reverse complemented separately (the order of a comma separated list of alleles is kept). Symbolic
alleles (starting with <, such as <DEL>), *, . and breakend alleles (containing [ or ]) are left
as they are. The alleles are empty for rows without a target.
Note
An indel that lifts to the - strand has its alleles reverse complemented allele by allele, but it is not
re-anchored or normalised. The anchor base of a VCF-style indel (the first base of REF and ALT) ends up at the end
of the allele instead of the start, and liftover_pos is the 1-based start of the lifted REF interval, its
lowest target coordinate. Left-align and normalise the result before comparing it with variants called on the target build.
With -var -aligned two columns are added after liftover_strand: liftover_aligned_left (I) and
liftover_aligned_right (I), the number of target bases left and right of liftover_pos that are in the same
gap-free alignment as the variant, as described for -aligned in LIFTOVER (Aligned extent). They are 0 for
every row that is not mapped, including passthrough rows. Use them to compare reference windows around the
variant between the builds without counting the bases a nearby build indel shifts.
-seg and -seg1: liftover_status, liftover_chrom, liftover_start (I), liftover_stop (I),
liftover_strand (S), liftover_chain_blocks (I, the number of chain blocks that overlap the segment, including
blocks that lose under -overlap best and filled gaps, see Equal gaps). The start and stop are in the same convention as the input: -seg
gives GOR half-open coordinates and -seg1 gives 1-based inclusive coordinates.
The liftover_strand column is + or - for the strand of the target relative to the source, and empty when
there is no target, and for passthrough rows.
With -diag the following columns are added after liftover_chain_blocks:
Column |
Type |
Description |
|---|---|---|
|
I |
The size of the source segment, last base minus first base, so a 100 base segment gives 99. |
|
I |
The distance between the two lifted-over endpoints when both endpoints lift to the same target contig, otherwise 0. A segment that is contained in one chain block has the same target size as source size. |
|
D |
|
|
S |
The contig of the lifted-over first base of the segment. Empty when the endpoint does not lift. |
|
I |
The 1-based position of the lifted-over first base. 0 when it does not lift. |
|
S |
The strand of the lift of the first base. Empty when it does not lift. |
|
S |
The same as |
|
I |
The same as |
|
S |
The same as |
A segment that one block contains reports its diagnostics through that block: the target size equals the source size,
the size ratio is 1.0 and the endpoints are lifted through the block, whether the status is mapped or
unsupported_contig (the block lifts to a contig that is not in -mapped). A passthrough row has the target size
equal to the source size, a ratio of 1.0 and empty endpoints ('', 0, ''). A junction (see Notes) has
empty endpoints. Endpoints that do not lift are also '', 0, ''.
The diagnostic columns are useful to see why a segment did not map, for example the size ratio of a
size_changed row, or which end of an inverted row went where.
Status values¶
Every row gets exactly one status. The statuses are tested in the order below, and the first one that applies is used.
-snp and -var:
passthrough: the source contig is in-passthrough. The row is copied through unchanged (chrom and pos are the input values, the strand is empty).unsupported_contig: the source contig is not in-mapped.ambiguous: with-overlap best, two best blocks have exactly the same score.unmapped: no block contains the position, and no filled gap or run of one chain does either (see Equal gaps). For-varthe block (or run) must contain the whole of REF. An empty REF is a junction (see Notes).unsupported_contig: the block lifts to a contig that is not in-mapped.mapped.
-seg and -seg1:
passthrough: the source contig is in-passthrough. The row is copied through unchanged.unsupported_contig: the source contig is not in-mapped.If one block (the best block under
-overlap best) contains the entire segment:ambiguous: two best blocks have exactly the same score.unsupported_contig: the block lifts to a contig that is not in-mapped.mapped.
This is the case for almost every segment, and the endpoint statuses below are then not considered.
Otherwise no block contains the segment as a whole, and the first and last base are lifted on their own:
unmapped: a segment that is a single base or a junction.ambiguous: the first or the last base is ambiguous, even when the other base does not lift.unmapped: neither the first nor the last base lifts.start_unmapped: only the first base does not lift.stop_unmapped: only the last base does not lift.split_contig: the two bases lift to different contigs.unsupported_contig: both lift to a contig that is not in-mapped.inverted: the two bases lift to opposite strands, or to the wrong order on the target.mapped_across_gap: both bases lift to the same contig and strand, in the same order, and the lifted size is within the tolerance of the source size, that is1 - tolerance < size_ratio < 1 + tolerance. The segment spans a gap between blocks that is small.size_changed: as above, but the size ratio is outside the tolerance.
Rows with a status that has a target (mapped, mapped_across_gap, size_changed and passthrough) have the
target location filled in. For the other statuses the location is empty ('', 0, 0, '').
Coordinate conventions¶
The input is converted to 0-based half-open coordinates before it is lifted, so a position p is [p-1,p), a
variant is [pos-1, pos-1+len(REF)) and a segment is [start,end) for -seg and [start-1,end) for
-seg1. This is the one place where -seg and -seg1 differ: the first is the GOR convention used for the
start and end columns of gor rows, the second a 1-based inclusive range as used by VCF positions, BED-like tools that
report start+1 and so on. The output uses the same convention as the input in each case, so with -seg1 the
lifted-over start is also 1-based and inclusive.
A segment is mapped only when a single chain block contains all of it. Adjacent blocks that together cover the
segment can give mapped_across_gap or size_changed (see the example below), and also inverted or
split_contig when the blocks lead to opposite strands or to different contigs.
Equal gaps¶
UCSC chains encode a run of substituted bases as a gap of the same length on both sides between two blocks (most are
one base). Such a gap is lifted as if the chain had been merged across it: a position in it is placed by its offset from
the block before it. Without this, a liftover with a chain that has not been merged reports those bases as
unmapped, while the opposite direction maps onto them.
A gap is filled only when all of these hold:
the blocks on either side are consecutive rows of the chain, with the same
qChrom,qStrandandqScore(a chain file has no chain id, andqScoreis the score of the whole chain);the gap has the same length on the source and the target side, and is not empty;
the gap is at most 50 bases long;
no other block lies in or reaches into the gap.
Longer equal gaps are blocks of different sequence rather than substituted bases, so they stay unmapped. The limit is
set with the Java property gor.liftover.gapfill.max (a whole number of bases, default 50; 0 fills no gap,
as -nogapfill does). It is read when the chain is loaded, and chains loaded with different limits are not shared.
An unequal gap (an indel), a gap longer than the limit, a gap between blocks of different chains and a gap that holds a block of another chain stay unmapped.
A filled gap is used only when no real block contains the input. Along a chain joined by filled gaps the offset does
not change, so a segment, -var REF or -bam read that runs from a block through one or more filled gaps into the
next blocks of the same chain is mapped through them, as it would be on a merged chain, and a junction on the edge
of a filled gap lifts too. Blocks that merely abut, with no filled gap between them, are not joined this way: a segment
across them is still classified by its endpoints (usually mapped_across_gap). liftover_chain_blocks counts the
filled gaps a segment covers as blocks, and LIFTOVER -all lists the run with the chain score when no block contains
the input.
-nogapfill turns all of this off, for the result of 5.16 and earlier.
Chain format¶
The chain is a GOR file (or nested query) with the columns Chrom, tStart, tEnd, qChrom, qStart,
qEnd, qStrand and qScore. The first two columns are the source contig and the 0-based start, so the file
must be sorted as a GOR file. Column names are matched case-insensitively.
The blocks are ungapped, 0-based half-open intervals:
tEnd - tStartmust equalqEnd - qStart.The target interval is always given on the forward strand of
qChrom, andqStrandis+or-, where-means that the source is reverse complemented relative to the target.qScoreis a number (it can be larger than an int). It is used to choose between overlapping blocks with-overlap best.The chain must be sorted genomically, that is by contig and then by
tStart, with every contig in one run. A file chain is validated one contig at a time, when a row on that contig is first lifted, and it is not scanned as a whole: a contig that reappears later in the file is not noticed, and which of its blocks are found is undefined. A nested chain is read as a whole, and a contig that reappears after another contig is rejected withLiftover chain <query>: contig <c> is not contiguous.A chain with no blocks (a header-only file or an empty nested query) is an error,
Liftover chain <path> has no blocks, reported on the first row that needs the chain.The chain is checked when it is loaded. A file chain is loaded one source contig at a time, the first time a row on that contig needs it, so an error in a block on a contig that no row uses is never reported (a nested chain is read and checked as a whole). A chain that is not sorted by
tStart, has a block withtEnd <= tStart, a gapped block, a negativeqStart, a missingtEnd,qChrom,qStartorqEnd, a lone-astEnd,qStartorqEnd, aqStrandother than+or-, an invalid score, or (with the default-overlap error) overlapping blocks is rejected with an error that starts withLiftover chain <path>. A negativetStartis the row position, so the GOR reader rejects it with its own error before the chain is checked.
Examples¶
The segment examples use this chain. It has two adjacent blocks on chr1, a block on the reverse strand, a block that
moves to another contig, two overlapping blocks (chr1 800-900 and chr1 850-950) and a block on chrM. The chain
has overlapping blocks, so -overlap best is needed to load it.
Chrom tStart tEnd qChrom qStart qEnd qStrand qScore
chr1 100 200 chr1 1100 1200 + 1000
chr1 200 300 chr1 1200 1300 + 1000
chr1 400 500 chr1 5000 5100 - 900
chr1 600 700 chr2 2100 2200 + 800
chr1 800 900 chr1 8800 8900 + 500
chr1 850 950 chr1 9850 9950 + 500
chrM 300 3100 chrM 298 3098 + 100
Segments in GOR coordinates, with -seg:
gor segments.gor | LIFTOVER_ANNOTATE chain.gor -seg -overlap best
Input rows chr1 120 180, chr1 150 250 and chr1 250 450 (named g01, g02 and g03) give:
Chrom Start End Name liftover_status liftover_chrom liftover_start liftover_stop liftover_strand liftover_chain_blocks
chr1 120 180 g01 mapped chr1 1120 1180 + 1
chr1 150 250 g02 mapped_across_gap chr1 1150 1250 + 2
chr1 250 450 g03 inverted 0 0 2
g01 lies inside the first block. g02 crosses from the first block into the second, so it is not contained in one block,
but both ends lift to the same contig and strand and the size is unchanged. g03 starts on the forward strand and ends
inside the reverse strand block, so it is inverted and has no target location.
Segments in 1-based inclusive coordinates, with -seg1. The same three segments written as 121-180,
151-250 and 251-450:
gor segments1.gor | LIFTOVER_ANNOTATE chain.gor -seg1 -overlap best
gives:
Chrom Start End Name liftover_status liftover_chrom liftover_start liftover_stop liftover_strand liftover_chain_blocks
chr1 121 180 g01 mapped chr1 1121 1180 + 1
chr1 151 250 g02 mapped_across_gap chr1 1151 1250 + 2
chr1 251 450 g03 inverted 0 0 2
The lifted start is 1-based here as well (1121 where -seg gave 1120).
A segment spanning a gap in the source that is closed in the target, with -diag. The chain has the blocks
chr1 100-200 -> 1000-1100 and chr1 210-310 -> 1100-1200, there is a gap of 10 bases between the blocks in the
source, and none in the target:
gor ... | LIFTOVER_ANNOTATE chain.gor -seg -diag
Chrom Start End Name liftover_status liftover_chrom liftover_start liftover_stop liftover_strand liftover_chain_blocks liftover_source_size liftover_target_size liftover_size_ratio liftover_start_chrom liftover_start_pos liftover_start_strand liftover_stop_chrom liftover_stop_pos liftover_stop_strand
chr1 120 180 in mapped chr1 1020 1080 + 1 59 59 1.0 chr1 1021 + chr1 1080 +
chr1 150 260 gap size_changed chr1 1050 1150 + 2 109 99 0.908256880733945 chr1 1051 + chr1 1150 +
The gap row has a size ratio of 99/109, outside the default tolerance of 0.05, so it is size_changed. With
-tolerance 0.1 it is mapped_across_gap.
A custom prefix for the new columns, with -snp:
gor snps.gor | LIFTOVER_ANNOTATE chain.gor -snp -prefix hg38_
adds the columns hg38_status, hg38_chrom, hg38_pos and hg38_strand. With -seg -prefix lifted_ -diag
the names are lifted_status, lifted_chrom, lifted_start, lifted_stop, lifted_strand,
lifted_chain_blocks, lifted_source_size, lifted_target_size, lifted_size_ratio,
lifted_start_chrom, lifted_start_pos, lifted_start_strand, lifted_stop_chrom, lifted_stop_pos and
lifted_stop_strand.
Keep the rows that mapped (and the chrM rows that were passed through), and, in the second query, fail on rows that cannot be lifted:
gor variants.gor | LIFTOVER_ANNOTATE hg19tohg38.gor -var -passthrough chrM | WHERE liftover_status = 'mapped' or liftover_status = 'passthrough'
gor variants.gor | LIFTOVER_ANNOTATE hg19tohg38.gor -var -failon unmapped,ambiguous
Notes¶
The rows are not relocated: the output has the same coordinates and order as the input, so the command is safe to use with
pgor. If you want the rows at their lifted-over positions, move the lifted-over columns into the first columns yourself and useSORT GENOMEafter that, because the relocated rows are no longer in order.The chain can be a nested query,
<(gor ...), for exampleLIFTOVER_ANNOTATE <(gor chain.gor | where qScore > 100) -snp. A nested chain is read once per pipe step and is validated like a file, but it is not cached across queries. A chain file is cached for the session, so repeated queries on the same file do not read it again. A nestednorquery is not accepted as a chain.To be cached, a chain file must have a unique source id in its file system or driver. A chain file without one is rejected with an error.
A zero-length interval is a junction between two bases:
-segwithend == start,-seg1withend == start - 1, and-varwith an empty REF. A junction maps only when it lies strictly inside a chain block, that is not at the edge of the block. The output of a junction is again a zero-length interval (-seg: start = stop,-seg1: stop = start - 1). An interval withend < start(end < start - 1for-seg1) is an error.The status of a segment comes from containment first. The endpoint statuses (
start_unmapped,split_contig,invertedand so on) are only used when no single block contains the whole segment.Errors in the data start with
LIFTOVER_ANNOTATE:and contain the contig and position of the row.In nor, the chromosome, position and, for segments, end columns must be given with
-chrom,-posand-end.