Start

Most of the PRGM-functions are called via syscalls. Some of the PRGM-functions are more or less worthless.

Such a function can be redirected by redirecting the corresponding syscall, like
0x0CB0: int PRGM_Send( char**program, short*opcode, TBCDvalue result[2] );
, which is called if Send() or Send38k() is called from out of a PRGM-program.

The replacement code has to be positioned at a fixed position, f. i. in sector 0x80220000, which is a unused sector throughout every fx-9860-OS. While developing some RAM could be used for convenience. F. i. onchip-RAM 0xA5600600 to 0xA5603A97 or 0x88040000 on 512k RAM-machines.
When using the onchip-RAM, at any rate the code must not exceed the allowed limits!

1. most important: make a backup of the OS of the calculator, where the PRGM extension should be installed.

2. find the address of syscall 0xCB0:
- read the pointer at fileoffset 0x0001007C: *(int*)0x8001007C
- subtract 0x80000000: *(int*)0x8001007C - 0x80000000
- now we have the fileoffset to the syscall-table.
- calculate the address of the syscall: ( *(int*)0x8001007C - 0x80000000 ) + 0xCB0*4
- subtract 0x80000000 again: ( ( *(int*)0x8001007C - 0x80000000 ) + 0xCB0*4 ) - 0x80000000
- now we have the fileoffset of syscall's 0xCB0 code, which has to be replaced.
- hint: the first four bytes must be 0x2F 0xE6 0x6E 0x53.

4. Now the replacement code must be assembled and bound. That requires the use of SHC/SHCPP, ASMSH and OPTLNK separately, t. i. not in the course of regular SDK-build. The SDK locates the code at 0x00300200 and inserts some initialization code, which would must not be inserted.

OK. I usually prefer small steps, if it comes to solve difficult problems, but here is the whole bunch for a start.
Sorry, if it seems to be overwhelming.
There is no doubt, that depending on different work environments, this scheme may not work at once and will lead to some complaints. But usually such inconveniences can be mended by minor adjustments. Do not despair. Simply ask instead.

You need a replacement function. You could write it in assembler, but more convenient is C/C++, of course. I prefer SHCPP because it issues a lot more warnings than SHC and so helps to prevent some avoidable errors. Lets call the source module CLIB.CPP.
The interface must be
int CLIB_entry( char**program, short*opcode, TBCDvalue result[2] );.
In the old documentation, the parameters program and opcode have been arranged in the wrong order! Sorry for that.
Here is the minimum frame for this function. It has to process the opcodes until an EndOfLine has been encountered, before returnning control to PRGM! Two syscalls are needed for that. What you do around this absolutely necessary frame is up to you. Return 1 if you want to report success. Return 0 if you want to report failure. In case of failure, you have to provide for some error information in TBCDvalue result[2]. If you do not provide for error information, the calc will crash!

// instant syscalls (fx-9860)
#define SCA 0xD201D002
#define SCB 0x422B0009
#define SCE 0x80010070
typedef void(*sc_vv)(void);
typedef int(*sc_iv)(void);

const unsigned int iPRGM_NextOpcode[] = { SCA, SCB, SCE, 0x0652 };
const unsigned int iPRGM_IsEndOfLine[] = { SCA, SCB, SCE, 0x06A6 };

//
void PRGM_NextOpcode( short*opcode, char**program ){
    (*(sc_vv)iPRGM_NextOpcode)();
}

//
int PRGM_IsEndOfLine( short*opcode ){
    return (*(sc_iv)iPRGM_IsEndOfLine)();
}
// instant syscalls end

typedef char TBCDvalue[12];

//
int CLIB_entry( char**program, short*opcode, TBCDvalue result[2] ){
int iresult = 1;

    while( 1 ){
        PRGM_NextOpcode( opcode, program );
        if ( PRGM_IsEndOfLine( opcode ) == 1 ) break;
// this is the part, where the parameters could be analyzed
    }

    return iresult;
}

!MESSAGE
!MESSAGE Executing Hitachi SHCPP phase
!MESSAGE

CPROG=CLIB

"$(CPROG).obj" : $(CPROG).cpp
"SHCPP" -subcommand=<<
$(CPROG).cpp
-I=...
-OB=$(CPROG).OBJ
-L=$(CPROG).LST
-CPU=sh3
-show=source
-size
-noinline
-chgincpath
-errorpath
-nodebug
-nomessage
<<

The SHCPP-option -I= must be set to the comma-separated list of INCLUDE-directories, if needed.

The path must contain the directory, where the CASIO SDK-binaries reside: usually ...\CASIO\fx-9860G SDK\OS\SH\BIN
The compiler needs the following environment variables:
SET SHC_LIB=usually ...\CASIO\fx-9860G SDK\OS\SH\BIN
SET SHC_TMP=some directory, where the user has write-rights (with CASIO SDK it is the directory called "..\debug")

Then a main assembler frame is required to ensure, that the entry of your replacement code will be at the top of the resulting binary. It could look like this (f. i. PRGM0CB0.ASM). CLIB_entry is the function in your CLIB.obj, which must be called, when the redirected syscall is invoked:

!MESSAGE
!MESSAGE Executing Hitachi ASMSH phase
!MESSAGE

PROG=PRGM0CB0

"$(PROG).obj" : $(PROG).asm
"ASMSH" -subcommand=<<
$(PROG).asm
-LIST=$(PROG).LST
-CPU=sh3
-endian=big
-round=zero
-denormalize=off
-debug
-literal=pool,branch,jump,return
-nologo
-chgincpath
-errorpath
<<

Finally PRGM0CB0.obj and CLIB.OBJ has to be linked together to give the resulting binary.

!MESSAGE
!MESSAGE Executing Hitachi OPTLNK phase
!MESSAGE

PROG=PRGM0CB0
CPROG=CLIB

"$(PROG).bin" : $(PROG).OBJ $(CPROG).obj
"OPTLNK" -subcommand=<<
noprelink
nomessage
list "$(PROG).map"
show symbol
nooptimize
start P_TOP,P,C,D/88070000
fsymbol P
nologo
input $(PROG).obj
input $(CPROG).obj
input setup.obj
output "$(PROG).abs"
-nomessage=1100
end
input "$(PROG).abs"
form binary
output "$(PROG).BIN"
exit
<<

setup.obj can be found in the CASIO SDK directory ...\CASIO\fx-9860G SDK\OS\FX\lib.
Depending on the complexity of CLIB.CPP, additional input- and/or library-declarations possibly must be included.

Before transferring the binary to the calculator the file PRGM0CB0.MAP should be inspected to ensure, that the entry is at 0x88070000. Maybe CLIB.LST and PRGM0CB0.LST should be inspected, too.

SuperH-based fx calculators fx-9860G, PRGM extension

SuperH-based fx calculators
fx-9860G, PRGM extension