Welcome to this forum vipiish,
You are right variable/modifier can't be embedded in regular expressions.
The "\10" case you report looks like an issue. I'll try to find an answer to it.
For the renaming you are trying to achieve I am not sure that a regular expression is the most appropriate.
Here is an expression that should do it (supposing that the hexa numbers are separated by a '-' and are the last part of the base name) :
%b(s/0x.*$//)0x%b[-2]("x","x0000000")(-8)-0x%b[-1]("x","x0000000")(-8).%e
Lets have a look at its different parts :
%b(s/0x.*$//) : extracts the text before the first hexadecimal number
0x%b[-2]("x","x0000000")(-8) : pads the first hexadecimal number to height digits
0x%b[-1]("x","x0000000")(-8) : pads the second hexadecimal number to height digits
.%e : adds the dot and the extension of the file
The "tricky" part is the padding. Lets detail the first one using the file name : File0xa0800-0x110000.dat
%b[-2] : will extract the last but one string supposing they are separated by '-'. In this case : File0xa0800
%b[-2]("x","x0000000") : will insert seven '0' just after the 'x'. In this case : File0x0000000a0800
%b[-2]("x","x0000000")(-8) : will extract the last height characters (the hexa number left paded). In this case : 000a0800
0x%b[-2]("x","x0000000")(-8) : will recreate the expected hexa number. In this case : 0x000a0800
I hope this will help,
Regards,
Remi