[Nasm-devel] A few rip-rel things

[Peter Johnson]

In going through the new rip relative stuff in NASM, and noticed a few
things:

1. It seems ugly to make an exception for just FS and GS to have an
exception to default-RIP-rel (and the current wording in the NASM manual
is confusing too)... wouldn't it be better to just turn off
default-RIP-rel for ALL segment registers? Particularly seeing as an es
override is ignored with a warning. I would think that if a user is using
*any* segment register, having default RIP-relative would be a surprise.


2. It appears handling of 64-bit movoffs values in default rel mode is
broken in NASM:

default rel
mov rax, [123456789abcdef0h]
mov rax, [qword 123456789abcdef0h]

both mov's generate 32-bit RIP-relative values with no warning. This
feels like a bug... but see #3 for more on what's the *real* issue here.


3. I believe that due to 64-bit movoffs forms being "special" (in allowing
a 64-bit displacement), the use of qword in the [] should force the
displacement size, not the address size (and likewise for dword, which
should force a 32-bit displacement size). Note displacement size can be
different than address size in 64-bit mode (okay, only for the movoffs
form, but consistency is good across the board)! Is there a current way
in NASM's world to force either a 32-bit or 64-bit displacement size (NOT
address size!) for mov rax, [...]?

Note that the default in 64-bit mode should be NOT to use the movoffs
form, as this is 4 bytes longer than the modrm-version. Plus a number of
object formats much prefer 32-bit relative offsets rather than the 64-bit
one (win64 for example chokes heavily on 64-bit relocs in the linker
stage, so you'll end up with broken behavior with the current NASM
output).

mov rax, [dword foo] makes the displacement 4 bytes, but generates an a32
prefix on what is still a movoffs form instruction. Not exactly what we
want.

This is why in yasm I use [dword ...] and [qword ...] to force the
displacement size, and a32 and a64 to force the address size, to separate
these two concepts.

So the question is, what should the following example output? (yasm
behavior in comments). Run it through the current NASM and think about
what makes more sense. :)

bits 64
val:

default abs

mov rax, [val] ; 48 8b ... (32-bit disp)
mov rax, [dword val] ; 48 8b ... (32-bit disp)
mov rax, [qword val] ; 48 a1 ... (64-bit disp)
a32 mov rax, [val] ; 67 48 a1 ... (32-bit disp)
a32 mov rax, [dword val] ; 67 48 a1 ... (32-bit disp)
a32 mov rax, [qword val] ; 67 48 a1 ... (32-bit disp)
; [this one is debatable on correctness,
; I chose in yasm to make a32 override]
a64 mov rax, [val] ; 48 8b ... (32-bit disp)
a64 mov rax, [dword val] ; 48 8b ... (32-bit disp)
a64 mov rax, [qword val] ; 48 a1 ... (64-bit disp)

mov rbx, [val] ; 48 8b ... (32-bit disp)
mov rbx, [dword val] ; 48 8b ... (32-bit disp)
mov rbx, [qword val] ; illegal (can't have 64-bit disp)
a32 mov rbx, [val] ; 67 48 8b ... (32-bit disp)
a32 mov rbx, [dword val] ; 67 48 8b ... (32-bit disp)
a32 mov rbx, [qword val] ; illegal (can't have 64-bit disp)
a64 mov rbx, [val] ; 48 8b ... (32-bit disp)
a64 mov rbx, [dword val] ; 48 8b ... (32-bit disp)
a64 mov rbx, [qword val] ; illegal (can't have 64-bit disp)

default rel

; yasm doesn't do this yet, but this is what I think makes sense

mov rax, [val] ; 48 8b ... (32-bit disp, RIP-rel)
mov rax, [dword val] ; 48 8b ... (32-bit disp, RIP-rel)
mov rax, [qword val] ; 48 a1 ... (64-bit disp, ABS)
a32 mov rax, [val] ; 67 48 8b ... (32-bit disp, RIP-rel)
a32 mov rax, [dword val] ; 67 48 8b ... (32-bit disp, RIP-rel)
a32 mov rax, [qword val] ; 67 48 8b ... (32-bit disp, RIP-rel)
; [this one is debatable on correctness,
; I chose in yasm to make a32 override]
a64 mov rax, [val] ; 48 8b ... (32-bit disp, RIP-rel)
a64 mov rax, [dword val] ; 48 8b ... (32-bit disp, RIP-rel)
a64 mov rax, [qword val] ; 48 a1 ... (64-bit disp, ABS)

mov rbx, [val] ; 48 8b ... (32-bit disp, RIP-rel)
mov rbx, [dword val] ; 48 8b ... (32-bit disp, RIP-rel)
mov rbx, [qword val] ; illegal (can't have 64-bit disp)
a32 mov rbx, [val] ; 67 48 8b ... (32-bit disp, RIP-rel)
a32 mov rbx, [dword val] ; 67 48 8b ... (32-bit disp, RIP-rel)
a32 mov rbx, [qword val] ; illegal (can't have 64-bit disp)
a64 mov rbx, [val] ; 48 8b ... (32-bit disp, RIP-rel)
a64 mov rbx, [dword val] ; 48 8b ... (32-bit disp, RIP-rel)
a64 mov rbx, [qword val] ; illegal (can't have 64-bit disp)

Peter

[Peter Johnson]

3. I believe that due to 64-bit movoffs forms being "special" (in allowing a
64-bit displacement), the use of qword in the [] should force the
displacement size, not the address size (and likewise for dword, which
should force a 32-bit displacement size). Note displacement size can be
different than address size in 64-bit mode (okay, only for the movoffs form,
but consistency is good across the board)! Is there a current way in NASM's
world to force either a 32-bit or 64-bit displacement size (NOT address
size!) for mov rax, [...]?
Note that the default in 64-bit mode should be NOT to use the movoffs form,
as this is 4 bytes longer than the modrm-version. Plus a number of object
formats much prefer 32-bit relative offsets rather than the 64-bit one
(win64 for example chokes heavily on 64-bit relocs in the linker stage, so
you'll end up with broken behavior with the current NASM output).
mov rax, [dword foo] makes the displacement 4 bytes, but generates an a32
prefix on what is still a movoffs form instruction. Not exactly what we
want.
This is why in yasm I use [dword ...] and [qword ...] to force the
displacement size, and a32 and a64 to force the address size, to separate
these two concepts.
So the question is, what should the following example output? (yasm behavior
in comments). Run it through the current NASM and think about what makes
more sense. :)
bits 64
default abs
mov rax, [val] ; 48 8b ... (32-bit disp)
mov rax, [dword val] ; 48 8b ... (32-bit disp)
mov rax, [qword val] ; 48 a1 ... (64-bit disp)
a32 mov rax, [val] ; 67 48 a1 ... (32-bit disp)
a32 mov rax, [dword val] ; 67 48 a1 ... (32-bit disp)
a32 mov rax, [qword val] ; 67 48 a1 ... (32-bit disp)
; [this one is debatable on correctness,
; I chose in yasm to make a32 override]
a64 mov rax, [val] ; 48 8b ... (32-bit disp)
a64 mov rax, [dword val] ; 48 8b ... (32-bit disp)
a64 mov rax, [qword val] ; 48 a1 ... (64-bit disp)
mov rbx, [val] ; 48 8b ... (32-bit disp)
mov rbx, [dword val] ; 48 8b ... (32-bit disp)
mov rbx, [qword val] ; illegal (can't have 64-bit disp)
a32 mov rbx, [val] ; 67 48 8b ... (32-bit disp)
a32 mov rbx, [dword val] ; 67 48 8b ... (32-bit disp)
a32 mov rbx, [qword val] ; illegal (can't have 64-bit disp)
a64 mov rbx, [val] ; 48 8b ... (32-bit disp)
a64 mov rbx, [dword val] ; 48 8b ... (32-bit disp)
a64 mov rbx, [qword val] ; illegal (can't have 64-bit disp)
default rel
; yasm doesn't do this yet, but this is what I think makes sense
mov rax, [val] ; 48 8b ... (32-bit disp, RIP-rel)
mov rax, [dword val] ; 48 8b ... (32-bit disp, RIP-rel)
mov rax, [qword val] ; 48 a1 ... (64-bit disp, ABS)
a32 mov rax, [val] ; 67 48 8b ... (32-bit disp, RIP-rel)
a32 mov rax, [dword val] ; 67 48 8b ... (32-bit disp, RIP-rel)
a32 mov rax, [qword val] ; 67 48 8b ... (32-bit disp, RIP-rel)
; [this one is debatable on correctness,
; I chose in yasm to make a32 override]
a64 mov rax, [val] ; 48 8b ... (32-bit disp, RIP-rel)
a64 mov rax, [dword val] ; 48 8b ... (32-bit disp, RIP-rel)
a64 mov rax, [qword val] ; 48 a1 ... (64-bit disp, ABS)
mov rbx, [val] ; 48 8b ... (32-bit disp, RIP-rel)
mov rbx, [dword val] ; 48 8b ... (32-bit disp, RIP-rel)
mov rbx, [qword val] ; illegal (can't have 64-bit disp)
a32 mov rbx, [val] ; 67 48 8b ... (32-bit disp, RIP-rel)
a32 mov rbx, [dword val] ; 67 48 8b ... (32-bit disp, RIP-rel)
a32 mov rbx, [qword val] ; illegal (can't have 64-bit disp)
a64 mov rbx, [val] ; 48 8b ... (32-bit disp, RIP-rel)
a64 mov rbx, [dword val] ; 48 8b ... (32-bit disp, RIP-rel)
a64 mov rbx, [qword val] ; illegal (can't have 64-bit disp)

And of course something similar applies to the recent 64-bit immediate
discussion as well...

; yasm's behavior in comments

mov rax, val ; 32-bit imm
mov rax, dword val ; 32-bit imm
mov rax, qword val ; 64-bit imm

mov rbx, val ; 32-bit imm
mov rbx, dword val ; 32-bit imm
mov rbx, qword val ; 64-bit imm

Peter

[H. Peter Anvin]

Post by Peter Johnson
In going through the new rip relative stuff in NASM, and noticed a few
1. It seems ugly to make an exception for just FS and GS to have an
exception to default-RIP-rel (and the current wording in the NASM manual
is confusing too)... wouldn't it be better to just turn off
default-RIP-rel for ALL segment registers? Particularly seeing as an es
override is ignored with a warning. I would think that if a user is using
*any* segment register, having default RIP-relative would be a surprise.

No, that really makes sense, as hideously non-orthogonal as it may be
(there are some ugly hacks in x86-64). nasm64developer pointed out to
me that on some CPUs, it is possible to enable limit checks for segment
registers even in 64-bit mode, but only FS and GS have bases.

Post by Peter Johnson
2. It appears handling of 64-bit movoffs values in default rel mode is
default rel
mov rax, [123456789abcdef0h]
mov rax, [qword 123456789abcdef0h]
both mov's generate 32-bit RIP-relative values with no warning. This
feels like a bug... but see #3 for more on what's the *real* issue here.

No, that's the right thing, because you don't know a priori that that
isn't a valid address in that range. [abs 123456789abcdef0h] should do
the movoffs form.

In current NASM, word, dword and qword on an addressing operand applies
to the address size whereas byte specifies the displacement size. This
is inconsistent, yes, but it is the established behaviour and would
definitely have to be maintained for non-64-bit code. Introducing new
behaviour for 64-bit mode would have to be balanced against the other
tradeoffs.

Post by Peter Johnson
3. I believe that due to 64-bit movoffs forms being "special" (in allowing
a 64-bit displacement), the use of qword in the [] should force the
displacement size, not the address size (and likewise for dword, which
should force a 32-bit displacement size). Note displacement size can be
different than address size in 64-bit mode (okay, only for the movoffs
form, but consistency is good across the board)! Is there a current way
in NASM's world to force either a 32-bit or 64-bit displacement size (NOT
address size!) for mov rax, [...]?

There isn't at the moment. See below, though.

However, I'm adamant that "mov rax,[abs 123456789abtcdef0h]" should use
the 64-bit form by default; the optimizer should be allow to reduce it
to 32 bits if the address fits.

Post by Peter Johnson
Note that the default in 64-bit mode should be NOT to use the movoffs
form, as this is 4 bytes longer than the modrm-version. Plus a number of
object formats much prefer 32-bit relative offsets rather than the 64-bit
one (win64 for example chokes heavily on 64-bit relocs in the linker
stage, so you'll end up with broken behavior with the current NASM
output).

That's bogus. That's what "default rel" is for.

The real question is how much value it is in the 32-bit displacement
forms as opposed to the a32 form. Remember, we're only talking absolute
addresses here (not relative addresses nor ); the 32-bit displacement
form can produce addresses in the range ­±2 GB whereas the a32 form
produces addresses in the range 0-4 GB. So the only consumers of the
former would be something like an OS kernel which doesn't use
RIP-relative addressing... a pretty rare beast.

Post by Peter Johnson
mov rax, [dword foo] makes the displacement 4 bytes, but generates an a32
prefix on what is still a movoffs form instruction. Not exactly what we
want.

No, I think that is exactly what we want.

I have mentioned in the past that I'd like to use the syntax:

mov rax,[abs a64 dword bluttan]

... to produce the 32-bit displacement form. Yes, it's heavy on syntax,
but it is such a rare corner case.

This isn't implemented yet, though, nor am I really convinced it's the
right thing.

Post by Peter Johnson
This is why in yasm I use [dword ...] and [qword ...] to force the
displacement size, and a32 and a64 to force the address size, to separate
these two concepts.

That's fine in many ways, but it is a much bigger departure from
historical NASM syntax. Even though we could do this in 64-bit mode
(ONLY!), I'm a bit leery of doing so for the benefit of one single
instruction.

-hpa

[Peter Johnson]

Post by H. Peter Anvin

Post by Peter Johnson
In going through the new rip relative stuff in NASM, and noticed a few
1. It seems ugly to make an exception for just FS and GS to have an
exception to default-RIP-rel (and the current wording in the NASM manual
is confusing too)... wouldn't it be better to just turn off
default-RIP-rel for ALL segment registers? Particularly seeing as an es
override is ignored with a warning. I would think that if a user is using
*any* segment register, having default RIP-relative would be a surprise.

No, that really makes sense, as hideously non-orthogonal as it may be
(there are some ugly hacks in x86-64). nasm64developer pointed out
to me that on some CPUs, it is possible to enable limit checks for segment
registers even in 64-bit mode, but only FS and GS have bases.

Okay, I see your point here, and nasm64developer just pointed out the limits
thing to me as well. Withdrawn. I would suggest fixing the wording in the
manual, however (it uses double-excepts, which is rather hard to follow).

Post by H. Peter Anvin

Post by Peter Johnson
2. It appears handling of 64-bit movoffs values in default rel mode is
default rel
mov rax, [123456789abcdef0h]
mov rax, [qword 123456789abcdef0h]
both mov's generate 32-bit RIP-relative values with no warning. This
feels like a bug... but see #3 for more on what's the *real* issue here.

No, that's the right thing, because you don't know a priori that that
isn't a valid address in that range. [abs 123456789abcdef0h] should
do the movoffs form.

How is silently truncating plain [123456789abcdef0h] *ever* a good thing? If
you've defaulted to rel, NASM has no way of knowing what RIP the program is
running at, and thus has no reasonable method of finding a 32-bit relative
address that will access that large 64-bit address. Instead, it just
truncates it to garbage; silently no less! At least give a warning, but I
would argue that it's obviously a 64-bit constant :).

Post by H. Peter Anvin

Post by Peter Johnson
3. I believe that due to 64-bit movoffs forms being "special" (in allowing
a 64-bit displacement), the use of qword in the [] should force the
displacement size, not the address size (and likewise for dword, which
should force a 32-bit displacement size). Note displacement size can be
different than address size in 64-bit mode (okay, only for the movoffs
form, but consistency is good across the board)! Is there a current way
in NASM's world to force either a 32-bit or 64-bit displacement size (NOT
address size!) for mov rax, [...]?

There isn't at the moment. See below, though.
However, I'm adamant that "mov rax,[abs 123456789abtcdef0h]" should use
the 64-bit form by default; the optimizer should be allow to reduce
it to 32 bits if the address fits.

What about for external labels? Constants get confusing to talk about because
they're obviously 32-bit or not, so it can be optimized either way. I feel
that 32-bit be the default for consistency reasons, as 64-bit is a special
case, and only works for rax, so it's inconsistent for 64-bit to be the
default for the rax case:

default abs
mov rbx, [var] ; 32-bit displacement
mov rcx, [var] ; 32-bit displacement
mov rax, [var] ; 64-bit displacement (!?)

Maybe this isn't a problem for binary targets, but it is a lurking surprise
for targets with relocation such as ELF and Win64, as rax uses may
mysteriously break. Compare to:

mov rbx, [var] ; 32-bit
mov rbx, [var] ; 32-bit
mov rax, [var] ; 32-bit
mov rax, [qword var] ; 64-bit displacement

(note GAS follows this rule relative to mov and movabs; the latter is required
for 64-bit displacement size, while the former uses 32-bit displacement size
in 64-bit mode... indicating you have to know to explicitly request the 64-bit
size)

And we don't want to be inconsistent for constants versus labels, so that
means mov rax,[xxx] should default to 32-bit for constants as well, until they
get larger than the 32-bit displacement size will allow (so still optimize,
just in the other direction).

Also, what about consistency with mov imm?

My proposed:
mov rbx, var ; 32-bit
mov rax, var ; 32-bit

mov rbx, qword var ; 64-bit
mov rax, qword var ; 64-bit

Post by H. Peter Anvin

Post by Peter Johnson
Note that the default in 64-bit mode should be NOT to use the movoffs
form, as this is 4 bytes longer than the modrm-version. Plus a number of
object formats much prefer 32-bit relative offsets rather than the 64-bit
one (win64 for example chokes heavily on 64-bit relocs in the linker
stage, so you'll end up with broken behavior with the current NASM
output).

That's bogus. That's what "default rel" is for.

Okay, but see my consistency argument above when you're talking about abs
(either explicit or implicit).

Post by H. Peter Anvin
The real question is how much value it is in the 32-bit displacement
forms as opposed to the a32 form. Remember, we're only talking absolute
addresses here (not relative addresses nor ); the 32-bit displacement
form can produce addresses in the range ­±2 GB whereas the a32 form
produces addresses in the range 0-4 GB. So the only consumers of the
former would be something like an OS kernel which doesn't use
RIP-relative addressing... a pretty rare beast.

Point taken. Since they're absolute addresses and the negative form is rare,
the A0-A3 forms should be fine to use. However, it's another inconsistency,
as it makes rax "special" (as rbx, etc. take signed values...).

Post by H. Peter Anvin

Post by Peter Johnson
mov rax, [dword foo] makes the displacement 4 bytes, but generates an a32
prefix on what is still a movoffs form instruction. Not exactly what we
want.

No, I think that is exactly what we want.
mov rax,[abs a64 dword bluttan]
... to produce the 32-bit displacement form. Yes, it's heavy on
syntax, but it is such a rare corner case.
This isn't implemented yet, though, nor am I really convinced it's
the right thing.

Post by Peter Johnson
This is why in yasm I use [dword ...] and [qword ...] to force the
displacement size, and a32 and a64 to force the address size, to separate
these two concepts.

(pulled from above for clarity of discussion)

Post by H. Peter Anvin
In current NASM, word, dword and qword on an addressing operand applies
to the address size whereas byte specifies the displacement size. This
is inconsistent, yes, but it is the established behaviour and would
definitely have to be maintained for non-64-bit code. Introducing
new behaviour for 64-bit mode would have to be balanced against the other
tradeoffs.

(end pull from above)

Post by H. Peter Anvin
That's fine in many ways, but it is a much bigger departure from
historical NASM syntax. Even though we could do this in 64-bit mode
(ONLY!), I'm a bit leery of doing so for the benefit of one single
instruction.

Actually, it depends on how you look at it. In 32-bit and 16-bit mode, the
address size and displacement size always matched, thus to get a 16-bit
displacement size you had to have a 16-bit address size. My interpretation
has always been that [word X] or [dword X] set the displacement size, and the
fact the address size was set to 16-bit and 32-bit respectively was an
artifact of getting achieving displacement size. This is confirmed by the
fact [byte X] is accepted, sets the displacement size, and doesn't set the
address size. This distinction between address size and displacement size
becomes important in 64-bit mode, where the address size is 64-bit but most
displacements are 32-bit. I think this interpretation is clearer as it
matches with how immediates are handled size-wise.

How often are people *really* wanting to override the address size to 32-bit
in 64-bit mode?

Peter

[H. Peter Anvin]

Before I start, I should point out that it looks like the run of having
time that I've had over the last two weeks seems to come to an end,
quickly. Unless someone else is willing to pick up the slack, I'm not
going to be able to do any kind of major rearchitecting.

Post by Peter Johnson
How is silently truncating plain [123456789abcdef0h] *ever* a good thing? If
you've defaulted to rel, NASM has no way of knowing what RIP the program is
running at, and thus has no reasonable method of finding a 32-bit relative
address that will access that large 64-bit address. Instead, it just
truncates it to garbage; silently no less! At least give a warning, but I
would argue that it's obviously a 64-bit constant :).

It might be obvious in a simple example, but when you throw in symbols
and the linker, things can go very complicated. The logical conclusion
of your argument would be that a program like:

bits 64
default rel

org 0000300000000000h
mov rax,[0000300020000000h]

... should get a warning, or magically change to the absolute form,
either of which is clearly wrong. Keep in mind that the origin might be
hidden in the linker -- if there is an overflow, it's the *linker's* job
to signal a warning (the ELF64 format, for example, has both signalling
and quiet relocations for this reason.) The binary backend counts as a
linker in this way.

Post by Peter Johnson

Post by H. Peter Anvin
However, I'm adamant that "mov rax,[abs 123456789abtcdef0h]" should use
the 64-bit form by default; the optimizer should be allow to reduce
it to 32 bits if the address fits.

What about for external labels? Constants get confusing to talk about because
they're obviously 32-bit or not, so it can be optimized either way. I feel
that 32-bit be the default for consistency reasons, as 64-bit is a special
case, and only works for rax, so it's inconsistent for 64-bit to be the
default abs
mov rbx, [var] ; 32-bit displacement
mov rcx, [var] ; 32-bit displacement
mov rax, [var] ; 64-bit displacement (!?)

Kind of funny, you were arguing, in effect, the exact opposite above.

Post by Peter Johnson
Maybe this isn't a problem for binary targets, but it is a lurking surprise
for targets with relocation such as ELF and Win64, as rax uses may
mov rbx, [var] ; 32-bit
mov rbx, [var] ; 32-bit
mov rax, [var] ; 32-bit
mov rax, [qword var] ; 64-bit displacement
(note GAS follows this rule relative to mov and movabs; the latter is required
for 64-bit displacement size, while the former uses 32-bit displacement size
in 64-bit mode... indicating you have to know to explicitly request the 64-bit
size)
And we don't want to be inconsistent for constants versus labels, so that
means mov rax,[xxx] should default to 32-bit for constants as well, until they
get larger than the 32-bit displacement size will allow (so still optimize,
just in the other direction).

I can accept the point that the 64-bit movabs form is special, and as
such it should have to have explicit qword decoration. I would like to
hear other people's opinion on this.

Post by Peter Johnson
Also, what about consistency with mov imm?
mov rbx, var ; 32-bit
mov rax, var ; 32-bit
mov rbx, qword var ; 64-bit
mov rax, qword var ; 64-bit

Again, I'm concerned about this one, but I'm willing to hear other
people's opinion.

Post by Peter Johnson
Actually, it depends on how you look at it. In 32-bit and 16-bit mode, the
address size and displacement size always matched, thus to get a 16-bit
displacement size you had to have a 16-bit address size. My interpretation
has always been that [word X] or [dword X] set the displacement size, and the
fact the address size was set to 16-bit and 32-bit respectively was an
artifact of getting achieving displacement size. This is confirmed by the
fact [byte X] is accepted, sets the displacement size, and doesn't set the
address size. This distinction between address size and displacement size
becomes important in 64-bit mode, where the address size is 64-bit but most
displacements are 32-bit. I think this interpretation is clearer as it
matches with how immediates are handled size-wise.

It's an interesting way to think about it, if nothing else. I think I
would need to let this stew for a bit.

Post by Peter Johnson
How often are people *really* wanting to override the address size to 32-bit
in 64-bit mode?

Probably more often than you'd think. Most user-space programs has text
and static data in the bottom 2 GB and assembly programmers like to do
weird stuff.

-hpa

[Peter Johnson]

Post by H. Peter Anvin
Before I start, I should point out that it looks like the run of having
time that I've had over the last two weeks seems to come to an end,
quickly. Unless someone else is willing to pick up the slack, I'm
not going to be able to do any kind of major rearchitecting.

Sorry to hear that, but I know the feeling. The reason I'm arguing these
points are twofold:

1) There's a fair amount of 64-bit yasm code out there (yasm has had AMD64
support since mid-2003), and I'm looking out for my users who might want to be
able to easily translate their existing code so it assembles with either yasm
or the new nasm. I also don't want nasm to go and create a subtle difference
between what looks in general to be identical syntaxes; you've managed to not
do this so far (e.g. differences in RIP-rel have been obvious and not breaking
compatibility in general), and I'd hate to see breakage like this happen at
the very end.

2) I don't really want to have to add a "nasm2" frontend to yasm that tweaks
these things to match NASM's interpretation. If I have to, I will, for
compatibilities' sake, but I don't think this is the preferred approach for
the community at large.

Post by H. Peter Anvin

Post by Peter Johnson
How is silently truncating plain [123456789abcdef0h] *ever* a good thing? If
you've defaulted to rel, NASM has no way of knowing what RIP the program is
running at, and thus has no reasonable method of finding a 32-bit relative
address that will access that large 64-bit address. Instead, it just
truncates it to garbage; silently no less! At least give a warning, but I
would argue that it's obviously a 64-bit constant :).

It might be obvious in a simple example, but when you throw in
symbols and the linker, things can go very complicated. The logical
bits 64
default rel
org 0000300000000000h
mov rax,[0000300020000000h]
... should get a warning, or magically change to the absolute form,
either of which is clearly wrong. Keep in mind that the origin
might be hidden in the linker -- if there is an overflow, it's the
*linker's* job to signal a warning (the ELF64 format, for example,
has both signalling and quiet relocations for this reason.) The
binary backend counts as a linker in this way.

Well, magically changing to the absolute form isn't *wrong* (it still accesses
the right address, after all), but I'll admit it's non-optimal, which to you
may feel like the same thing? :) And maybe you're right, a warning can't be
sensibly generated here; but an absolute form can be! To my mind, non-optimal
output is okay; broken output isn't. And NASM's current output is broken in
this case; the mov gets a garbage RIP-relative address (well, in your exact
example above it doesn't, but if you change the org it does:

default rel
org 0000300010000000h
mov rax,[0000300020000000h] ; look at what NASM generates right now; it
generates code that doesn't access the location the assembly code says it
accesses...

"Someday" we could make it smarter optimization wise to know that this is
reachable with RIP-relative thanks to the ORG, but in the meantime it needs to
not generate incorrect code.

Post by H. Peter Anvin

Post by Peter Johnson

Post by H. Peter Anvin
However, I'm adamant that "mov rax,[abs 123456789abtcdef0h]" should use
the 64-bit form by default; the optimizer should be allow to reduce
it to 32 bits if the address fits.

What about for external labels? Constants get confusing to talk about because
they're obviously 32-bit or not, so it can be optimized either way. I feel
that 32-bit be the default for consistency reasons, as 64-bit is a special
case, and only works for rax, so it's inconsistent for 64-bit to be the
default abs
mov rbx, [var] ; 32-bit displacement
mov rcx, [var] ; 32-bit displacement
mov rax, [var] ; 64-bit displacement (!?)

Kind of funny, you were arguing, in effect, the exact opposite above.

I don't believe so, as I'm talking about default abs in this case, not default
rel as I was above. When you're in default abs mode, you should expect
non-rel addresses to be truncated; in default rel mode, you should expect the
assembler should use REL where it's posssible to do so, ABS if it's not
possible to reach with REL (or not determinable if it could be), and warn (or
error) if it's not possible to do either or is ambiguous.

Post by H. Peter Anvin

Post by Peter Johnson
Maybe this isn't a problem for binary targets, but it is a lurking surprise
for targets with relocation such as ELF and Win64, as rax uses may
mov rbx, [var] ; 32-bit
mov rbx, [var] ; 32-bit
mov rax, [var] ; 32-bit
mov rax, [qword var] ; 64-bit displacement
(note GAS follows this rule relative to mov and movabs; the latter is required
for 64-bit displacement size, while the former uses 32-bit displacement size
in 64-bit mode... indicating you have to know to explicitly request the 64-bit
size)
And we don't want to be inconsistent for constants versus labels, so that
means mov rax,[xxx] should default to 32-bit for constants as well, until they
get larger than the 32-bit displacement size will allow (so still optimize,
just in the other direction).

I can accept the point that the 64-bit movabs form is special, and as
such it should have to have explicit qword decoration. I would like
to hear other people's opinion on this.

Post by Peter Johnson
Also, what about consistency with mov imm?
mov rbx, var ; 32-bit
mov rax, var ; 32-bit
mov rbx, qword var ; 64-bit
mov rax, qword var ; 64-bit

Again, I'm concerned about this one, but I'm willing to hear other
people's opinion.

Post by Peter Johnson
Actually, it depends on how you look at it. In 32-bit and 16-bit mode, the
address size and displacement size always matched, thus to get a 16-bit
displacement size you had to have a 16-bit address size. My interpretation
has always been that [word X] or [dword X] set the displacement size, and the
fact the address size was set to 16-bit and 32-bit respectively was an
artifact of getting achieving displacement size. This is confirmed by the
fact [byte X] is accepted, sets the displacement size, and doesn't set the
address size. This distinction between address size and displacement size
becomes important in 64-bit mode, where the address size is 64-bit but most
displacements are 32-bit. I think this interpretation is clearer as it
matches with how immediates are handled size-wise.

It's an interesting way to think about it, if nothing else. I think
I would need to let this stew for a bit.

Post by Peter Johnson
How often are people *really* wanting to override the address size to 32-bit
in 64-bit mode?

Probably more often than you'd think. Most user-space programs has text
and static data in the bottom 2 GB and assembly programmers like to
do weird stuff.

Okay, but I guess my counter-argument to that is still that "safe" programming
in 64-bit mode generally means that you (the coder) want to be very explicit
when you're truncating the address to 32-bit, particularly if you're running
default rel as I suspect a lot of people will be. Just a [dword X] doesn't
feel quite explicit enough to me.

Peter

[H. Peter Anvin]

Post by Peter Johnson
Well, magically changing to the absolute form isn't *wrong* (it still accesses
the right address, after all), but I'll admit it's non-optimal, which to you
may feel like the same thing? :)

No, it's wrong, as it changes the properties of the code, e.g. with
respect to relocation. Invisibly changing between absolute and relative
is a red line is my book.

Post by Peter Johnson
default rel
org 0000300010000000h
mov rax,[0000300020000000h] ; look at what NASM generates right now; it
generates code that doesn't access the location the assembly code says it
accesses...
"Someday" we could make it smarter optimization wise to know that this is
reachable with RIP-relative thanks to the ORG, but in the meantime it needs to
not generate incorrect code.

No, the right thing here is for the linker -- in this case, the outbin
backend -- to issue the warning.

Post by Peter Johnson
Okay, but I guess my counter-argument to that is still that "safe" programming
in 64-bit mode generally means that you (the coder) want to be very explicit
when you're truncating the address to 32-bit, particularly if you're running
default rel as I suspect a lot of people will be. Just a [dword X] doesn't
feel quite explicit enough to me.

The more I'm thinking about it, the more I think you're probably right,
in combination with proper documentation.

The key to making a sensible pattern here is to note that the qword
forms *are* special, the programmer needs to know where they can be
used, and that they therefore should be called out by the programmer.

In other words:

mov rax,[foo] ; 32-bit displacement
mov rax,[qword foo] ; 64-bit memoffs
mov rax,bar ; 32-bit value
mov rax,qword bar ; 64-bit value

The latter, however, is potentially very ugly for the programmer, since
it is "less special" -- works with any register and people feel like:

mov rcx,0123456789abcdefh

should "just work". On the other hand, you could argue (almost) equally
well that:

add rcx,0123456789abcdefh

should "just work", and that the assembler should figure out some way to
deal with it, like:

push rax
mov rax,0123456789abcdefh
add rcx,rax
pop rax

This may seem absurd, but there are people actually subscribing to that
school of assembly programming -- MIPS actually reserves a register for
the assembler to use in breaking up impossible instruction patterns!

I'm really of two minds on this one; either solution seems poor. On one
hand treating addresses and operands differently is really ugly, on the
other hand, if you wanted to set a 32-bit operand you would simply do:

mov eax,01234567h

... since eax will automatically zero-extend into rax. Unlike addresses
and displacements, there isn't any zero- versus sign-extend ambiguity.
The existence of a natural syntax for both sizes imply that different
treatment might be in order.

At this time, I think I'm leaning toward the following:

- Operands default to qword automatically;
- Addresses will not use the 64-bit form unless explicitly
declared qword;
- dword will set the displacement size, not the address size;
- a32/a64 will be supported inside brackets;
- no promotion from rel to abs or vice versa under any
circumstances.

Inside NASM, this means adding a notion of address size as distinct from
the displacement size. I used to think it should have been a global
property, but now I'm vaguely leaning toward adding it to the operand
structure like the existing address (displacement) size.

Dealing with o16/o32/o64 as anything but a passive prefix would be good,
too.

-hpa

[Peter Johnson]

Post by H. Peter Anvin

Post by Peter Johnson
Well, magically changing to the absolute form isn't *wrong* (it still accesses
the right address, after all), but I'll admit it's non-optimal, which to you
may feel like the same thing? :)

No, it's wrong, as it changes the properties of the code, e.g. with
respect to relocation. Invisibly changing between absolute and relative
is a red line is my book.

I'm okay with this, as long as...

Post by H. Peter Anvin

Post by Peter Johnson
default rel
org 0000300010000000h
mov rax,[0000300020000000h] ; look at what NASM generates right now; it
generates code that doesn't access the location the assembly code says it
accesses...
"Someday" we could make it smarter optimization wise to know that this is
reachable with RIP-relative thanks to the ORG, but in the meantime it needs to
not generate incorrect code.

No, the right thing here is for the linker -- in this case, the
outbin backend -- to issue the warning.

This indeed does generate a warning (as nasm currently doesn't generate one
for outbin backend). I'm just not happy with generating likely incorrect code
AND having no warning. Again, my complaint at the start was that I didn't get
a warning from outbin.

Post by H. Peter Anvin

Post by Peter Johnson
Okay, but I guess my counter-argument to that is still that "safe" programming
in 64-bit mode generally means that you (the coder) want to be very explicit
when you're truncating the address to 32-bit, particularly if you're running
default rel as I suspect a lot of people will be. Just a [dword X] doesn't
feel quite explicit enough to me.

The more I'm thinking about it, the more I think you're probably
right, in combination with proper documentation.
The key to making a sensible pattern here is to note that the qword
forms *are* special, the programmer needs to know where they can be
used, and that they therefore should be called out by the programmer.
mov rax,[foo] ; 32-bit displacement
mov rax,[qword foo] ; 64-bit memoffs
mov rax,bar ; 32-bit value
mov rax,qword bar ; 64-bit value

I'm in complete agreement with this approach.

Post by H. Peter Anvin
The latter, however, is potentially very ugly for the programmer, since
mov rcx,0123456789abcdefh
should "just work". On the other hand, you could argue (almost) equally
add rcx,0123456789abcdefh
should "just work", and that the assembler should figure out some
push rax
mov rax,0123456789abcdefh
add rcx,rax
pop rax
This may seem absurd, but there are people actually subscribing to that
school of assembly programming -- MIPS actually reserves a register for
the assembler to use in breaking up impossible instruction patterns!
I'm really of two minds on this one; either solution seems poor. On
one hand treating addresses and operands differently is really ugly,
on the other hand, if you wanted to set a 32-bit operand you would
mov eax,01234567h
... since eax will automatically zero-extend into rax. Unlike addresses
and displacements, there isn't any zero- versus sign-extend
ambiguity. The existence of a natural syntax for both sizes imply
that different treatment might be in order.
- Operands default to qword automatically;

This seems like it's in conflict with your example and conclusions above,
unless this was a typo and you meant dword? Trying to follow your reasoning

Post by H. Peter Anvin
mov rax,[foo] ; 32-bit displacement
mov rax,[qword foo] ; 64-bit memoffs
mov rax,bar ; 32-bit value
mov rax,qword bar ; 64-bit value

(generating a warning for non-qword cases if bar is an equ value that can't
fit into 32 bits, of course)

Post by H. Peter Anvin
- Addresses will not use the 64-bit form unless explicitly
declared qword;

Yes!

Post by H. Peter Anvin
- dword will set the displacement size, not the address size;

Yes!

Post by H. Peter Anvin
- a32/a64 will be supported inside brackets;

Okay. A little syntax sugar is fine with me.

Post by H. Peter Anvin
- no promotion from rel to abs or vice versa under any
circumstances.

Based on value, I assume you mean? We still have [foo+rax] and [fs:foo] which
switch to abs, and [rel foo] which switches to rel. :) I'm okay with this,
given that we generate a warning when it's not knowably correct.

Post by H. Peter Anvin
Inside NASM, this means adding a notion of address size as distinct from
the displacement size. I used to think it should have been a global
property, but now I'm vaguely leaning toward adding it to the operand
structure like the existing address (displacement) size.
Dealing with o16/o32/o64 as anything but a passive prefix would be
good, too.

It sounds like we're mostly on the same page now.

Thanks,
Peter

[H. Peter Anvin]

Post by Peter Johnson

Post by H. Peter Anvin
- Operands default to qword automatically;

This seems like it's in conflict with your example and conclusions above,
unless this was a typo and you meant dword? Trying to follow your reasoning

Post by H. Peter Anvin
mov rax,[foo] ; 32-bit displacement
mov rax,[qword foo] ; 64-bit memoffs
mov rax,bar ; 32-bit value
mov rax,qword bar ; 64-bit value

(generating a warning for non-qword cases if bar is an equ value that can't
fit into 32 bits, of course)

No, I meant qword. The logic being that if the programmer had wanted a
32-bit value in the third case, they would have written "mov eax,bar".

It's an inconsistency, for sure, but given the particulars of the
instruction set, I think it's the best way to meet programmer expectations.

Post by Peter Johnson

Post by H. Peter Anvin
- Addresses will not use the 64-bit form unless explicitly
declared qword;

Yes!

Post by H. Peter Anvin
- dword will set the displacement size, not the address size;

Yes!

Post by H. Peter Anvin
- a32/a64 will be supported inside brackets;

Okay. A little syntax sugar is fine with me.

Post by H. Peter Anvin
- no promotion from rel to abs or vice versa under any
circumstances.

Based on value, I assume you mean? We still have [foo+rax] and [fs:foo] which
switch to abs, and [rel foo] which switches to rel. :) I'm okay with this,
given that we generate a warning when it's not knowably correct.

Yes, based on value.

-hpa

[Peter Johnson]

Post by H. Peter Anvin

Post by Peter Johnson

Post by H. Peter Anvin
- Operands default to qword automatically;

This seems like it's in conflict with your example and conclusions above,
unless this was a typo and you meant dword? Trying to follow your reasoning

Post by H. Peter Anvin
mov rax,[foo] ; 32-bit displacement
mov rax,[qword foo] ; 64-bit memoffs
mov rax,bar ; 32-bit value
mov rax,qword bar ; 64-bit value

(generating a warning for non-qword cases if bar is an equ value that can't
fit into 32 bits, of course)

No, I meant qword. The logic being that if the programmer had
wanted a 32-bit value in the third case, they would have written
"mov eax,bar".
It's an inconsistency, for sure, but given the particulars of the
instruction set, I think it's the best way to meet programmer expectations.

Okay, I need to think over this. I see where you're coming from, but I also
dislike inconsistencies (I know, near-impossibility given the state of x86,
but hey, I can try..) :)

Peter

[Peter Johnson]

Post by H. Peter Anvin

Post by Peter Johnson

Post by H. Peter Anvin
- Operands default to qword automatically;

This seems like it's in conflict with your example and conclusions above,
unless this was a typo and you meant dword? Trying to follow your reasoning

Post by H. Peter Anvin
mov rax,[foo] ; 32-bit displacement
mov rax,[qword foo] ; 64-bit memoffs
mov rax,bar ; 32-bit value
mov rax,qword bar ; 64-bit value

(generating a warning for non-qword cases if bar is an equ value that can't
fit into 32 bits, of course)

No, I meant qword. The logic being that if the programmer had wanted a
32-bit value in the third case, they would have written "mov eax,bar".
It's an inconsistency, for sure, but given the particulars of the
instruction set, I think it's the best way to meet programmer expectations.

Again, I see where you're coming from, but the counter-examples are
numerous, seeing as everwhere else in the instruction set, even with
reg64, the value is a 32-bit signed value by default:

mov reg64, bar -- (using the C7 encoding) == 32-bit signed immediate
mov qword [foo], bar == 32-bit signed immediate
<anything else> rm64, bar == 32-bit signed immediate
mov reg64, [bar] == 32-bit signed displacement
<anything else> reg64, [bar] == 32-bit signed displacement

I see your point that locally to mov, it makes sense to make mov reg64,
imm auto to a 64-bit immediate, as there is the alternative of mov reg32,
imm, but in the larger context of the instruction set this is a very
special case, as *everything* else treats an unsized value as a 32-bit
signed value in a 64-bit context, and thus I would say this special-case
should be recognized as such with an explicit "qword". In other words,
the special behavior of mov reg64, imm in this regard would be surprising
to the programmer since everything else autos to a 32-bit signed imm.

Peter

[Peter Johnson]

Post by Peter Johnson

Post by H. Peter Anvin

Post by Peter Johnson

Post by H. Peter Anvin
- Operands default to qword automatically;

This seems like it's in conflict with your example and conclusions above,
unless this was a typo and you meant dword? Trying to follow your reasoning

Post by H. Peter Anvin
mov rax,[foo] ; 32-bit displacement
mov rax,[qword foo] ; 64-bit memoffs
mov rax,bar ; 32-bit value
mov rax,qword bar ; 64-bit value

(generating a warning for non-qword cases if bar is an equ value that can't
fit into 32 bits, of course)

No, I meant qword. The logic being that if the programmer had wanted a
32-bit value in the third case, they would have written "mov eax,bar".
It's an inconsistency, for sure, but given the particulars of the
instruction set, I think it's the best way to meet programmer expectations.

Again, I see where you're coming from, but the counter-examples are
numerous, seeing as everwhere else in the instruction set, even with
mov reg64, bar -- (using the C7 encoding) == 32-bit signed immediate
mov qword [foo], bar == 32-bit signed immediate
<anything else> rm64, bar == 32-bit signed immediate
mov reg64, [bar] == 32-bit signed displacement
<anything else> reg64, [bar] == 32-bit signed displacement
I see your point that locally to mov, it makes sense to make mov reg64,
imm auto to a 64-bit immediate, as there is the alternative of mov reg32,
imm, but in the larger context of the instruction set this is a very
special case, as *everything* else treats an unsized value as a 32-bit
signed value in a 64-bit context, and thus I would say this special-case
should be recognized as such with an explicit "qword". In other words,
the special behavior of mov reg64, imm in this regard would be surprising
to the programmer since everything else autos to a 32-bit signed imm.

Let me clarify my thinking some more with some code examples. I don't
think that there should be a problem with automatically upsizing
immediates if they're non-relocatable (unlike ea's which indeed have the
rel/abs issue which argues for not forcing from rel to 64-bit abs) for
ease-of-use reasons, but I think it's important that externs/relocatables
don't magically upsize to 64-bit for that special case. Thus:

bits 64
default abs
extern label1
label2:

mov rax, 0x1000 ; 32-bit imm
mov rax, 0x1122334455667788 ; 64-bit imm (larger than 32-bit)
mov rax, label1 ; 32-bit imm <- not 64-bit, to match below
mov rax, label2 ; 32-bit imm <- not 64-bit, to match below
mov rax, qword 0x1000 ; 64-bit imm
mov rax, qword label1 ; 64-bit imm
mov rax, qword label2 ; 64-bit imm

; note parity here with defaults in imm case
mov qword [rax], 0x1000 ; 32-bit imm
mov qword [rax], 0x1122334455667788 ; 32-bit imm, overflow warning
mov qword [rax], label1 ; 32-bit imm (matches default above)
mov qword [rax], label2 ; 32-bit imm (matches default above)

add rax, 0x1000 ; 32-bit imm
add rax, 0x1122334455667788 ; 32-bit imm, overflow warning
add rax, label1 ; 32-bit imm (matches default above)
add rax, label2 ; 32-bit imm (matches default above)

; note parity here with imm case
mov [0x1000], rax ; 32-bit disp
mov [abs 0x1122334455667788], rax ; 64-bit disp
; (I think that in abs mode this should auto-upsize to 64-bit to match
; imm auto-upsize; agreed that it shouldn't in default rel mode)
mov [label1], rax ; 32-bit disp
mov [label2], rax ; 32-bit disp
mov [qword 0x1000], rax ; 64-bit disp
mov [qword label1], rax ; 64-bit disp
mov [qword label2], rax ; 64-bit disp

Peter

[H. Peter Anvin]

Post by Peter Johnson
Let me clarify my thinking some more with some code examples. I don't
think that there should be a problem with automatically upsizing
immediates if they're non-relocatable (unlike ea's which indeed have the
rel/abs issue which argues for not forcing from rel to 64-bit abs) for
ease-of-use reasons, but I think it's important that externs/relocatables

Now *that* is an inconsistency I really don't want to see. Instruction
selection depending on label or not (other than inhibiting
optimization)... no please.

Again, if the programmer wanted a 32-bit set, they'd simply write:

mov eax, foo

... instead of ...

mov rax, foo

-hpa

[Peter Johnson]

Post by H. Peter Anvin

Post by Peter Johnson
Let me clarify my thinking some more with some code examples. I don't
think that there should be a problem with automatically upsizing
immediates if they're non-relocatable (unlike ea's which indeed have the
rel/abs issue which argues for not forcing from rel to 64-bit abs) for
ease-of-use reasons, but I think it's important that externs/relocatables

Now *that* is an inconsistency I really don't want to see. Instruction
selection depending on label or not (other than inhibiting
optimization)... no please.
mov eax, foo
... instead of ...
mov rax, foo

I find this to be an uglier inconsistency:

mov qword [rax], foo ; 32-bit imm
<anything else> rax, foo ; 32-bit imm
mov rax, foo ; 64-bit imm !?

If you don't want auto-upsizing on raw values, fine, but I think this
inconsistency is ugly. x86 is complex enough that we want to find
consistency where we can.

Peter

[H. Peter Anvin]

Post by Peter Johnson

Post by H. Peter Anvin
mov eax, foo
... instead of ...
mov rax, foo

mov qword [rax], foo ; 32-bit imm
<anything else> rax, foo ; 32-bit imm
mov rax, foo ; 64-bit imm !?
If you don't want auto-upsizing on raw values, fine, but I think this
inconsistency is ugly. x86 is complex enough that we want to find
consistency where we can.

I'm sorry you disagree. I have heard your opinion by now.

-hpa