Will it be possible for someone to provide me a working example of reqExecutions? I am having a hard time with the ewrapper and callback mechanism. After searching google for working examples, I could not get my hands on anything that would simply work. Please note that I am not a programmer and that is why having a hard time getting my head wrapped around ewrapper and callback.
Before answering this question, I feel I should emphasize the disclaimer given at the very start of the IBrokers documentation:
This software is in no way affiliated, endorsed, or approved by Interactive Brokers or any of its affiliates. It comes with absolutely no warranty and should not be used in actual trading unless the user can read and understand the source.
So it looks like this package is designed and maintained by independent programmers who may or may not have a good tie-in with official IB API development now or in the future.
Further to the above, I looked at a lot of the package source, and it's fairly incomplete, vis-à-vis the actual IB API source. In fact, you've stumbled upon one of the strands of incompleteness; in the IBrokers Reference Card it says:
Executions
Returns execution details in a twsExecution object. This method is currently only implemented as a request, with no built-in mechanism to manage response data apart from it being discarded.
(Note: You can access the reference card with IBrokersRef(), if you've configured your options()$pdfviewer. If not, you can just open the PDF manually; run system.file('doc/IBrokersREFCARD.pdf',package='IBrokers') to get its location.)
So, the bottom line is, be careful with this package; you shouldn't rely on it for real trading unless you really know what you're doing.
Taking a look at the actual reqExecutions() package function, we have:
function (twsconn, reqId = "0", ExecutionFilter)
{
if (!is.twsConnection(twsconn))
stop("invalid 'twsConnection' object")
con <- twsconn[[1]]
VERSION <- "3"
outgoing <- c(.twsOutgoingMSG$REQ_EXECUTIONS, VERSION, as.character(reqId),
ExecutionFilter$clientId, ExecutionFilter$acctCode, ExecutionFilter$time,
ExecutionFilter$symbol, ExecutionFilter$secType, ExecutionFilter$exchange,
ExecutionFilter$side)
writeBin(outgoing, con)
}
To summarize the above, it:
is.twsConnection(), it just checks that it inherits from twsConnection or twsconn. You can create such a connection with twsConnect(). It's a plain R environment internally, classed as twsconn.`[[`() function for the twsconn class. If you look into IBrokers:::`[[.twsconn`, you see it just returns the twsconn$conn environment entry.writeBin()): a request type enumeration, a request version, a request identifier (which could be used to differentiate multiple simultaneous requests of the same type), and then 7 filter fields. (Unfortunately, it requires the caller to fully specify all filter fields.) It then returns immediately without waiting for a response.Contrast the above with a fully-implemented request function, reqCurrentTime():
function (twsconn)
{
.reqCurrentTime(twsconn)
con <- twsconn[[1]]
e_current_time <- eWrapper()
e_current_time$currentTime <- function(curMsg, msg, timestamp,
file, ...) {
msg[2]
}
while (isConnected(twsconn)) {
socketSelect(list(con), FALSE, NULL)
curMsg <- readBin(con, character(), 1)
currentTime <- processMsg(curMsg, con, eWrapper = e_current_time,
twsconn = twsconn, timestamp = NULL, file = "")
if (curMsg == .twsIncomingMSG$CURRENT_TIME)
break
}
structure(as.numeric(currentTime), class = c("POSIXt", "POSIXct"))
}
This:
.reqCurrentTime() to send the actual request, similar to what reqExecutions() does. I won't include it here, but you can view its body with IBrokers:::.reqCurrentTime.eWrapper() which it strangely names e_current_time.currentTime() on the callback list object. The handler simply returns the second field, msg[2], which represents the server's idea of the current time, encoded as a Unix epoch value (seconds since 1970-01-01).curMsg, which is a code representing the message type, and calls processMsg() on it. This is another function provided by the IBrokers package. This is the function that actually switches on the message code and calls the appropriate callback function on e_current_time, passing it curMsg as well as the code-specific trailing fields, decoded via a call to readBin() (not shown above; see processMsg for the code).msg[2], the second field after the message code) into currentTime.processMsg() actually triggered a default handler which doesn't do anything useful, and the currentTime value would be ignored.)currentTime value. All that's really required here is classing it as POSIXt and POSIXct, since the POSIXct type is conveniently implemented by storing the Unix epoch time to represent a date/time point.So, as you can see, reqCurrentTime() is doing a lot more than reqExecutions(). In its current form, reqExecutions() doesn't do anything to process the response, so it's not really useful at all.
Since I'm familiar with the IB API, I was able to fill in the missing functionality, which I present below.
As a side note, I referenced the C++ API source code which is available from https://www.interactivebrokers.com/en/index.php?f=5041; the official API source can be taken as authoritative with respect to how the client needs to interact with the socket. For example, you can view the EClient::reqExecutions() function in /TWS API/source/CppClient/client/EClient.cpp to see how it encodes request fields onto the socket, and similarly you can view the EDecoder::processExecutionDataMsg() function in the /TWS API/source/CppClient/client/EDecoder.cpp file to see how it decodes the response from the server. Basically, it uses some C preprocessor macros (ENCODE_FIELD() and DECODE_FIELD()) which expand to a call to some C++ template functions for encoding (template<class T> void EClient::EncodeField(std::ostream& os, T value)) and C++ overloaded functions for decoding (bool EDecoder::DecodeField(bool/int/long/double/std::string& doubleValue, const char*& ptr, const char* endPtr)) which ultimately use the C++ streaming operators to stream primitive fields to the socket std::ostream followed by a NUL for encoding, and call atoi(), atof(), or std::string::operator=() for decoding right from the socket buffer.
I also recommend looking into the official API documentation at https://www.interactivebrokers.com/en/software/api/api.htm.
Firstly, notice that IBrokers provides functions like twsContract() and twsOrder() to allow constructing TWS-specific data objects. There's no twsExecution() function, so I wrote my own, following the same style of object construction, including attaching a twsExecution S3 class. I also wrote a print.twsExecution() function so twsExecution objects would print the same way as the others, which basically means str(unclass(x)).
Secondly, I wrote my own replacement for reqExecutions() called reqExecutions2() which encodes the request data onto the socket as per reqExecutions(), and then overrides the response handler and iterates on the socket waiting for response messages, similar to reqCurrentTime(). I was a little bit more verbose with the code, as I tried to follow the C++ algorithm as closely as possible, including its request version checks on response handling to conditionally take certain fields off the socket. I also included monitoring for error messages from the server, so that the while loop automatically breaks and the function returns on errors. reqExecutions2() returns all response records as a list, where each component is a nested list of reqId, contract, and execution components. This follows the execDetails() callback design in the official API; see https://www.interactivebrokers.com/en/software/api/api.htm (C++ -> Class EWrapper Functions -> Executions -> execDetails()).
Lastly, for convenience I wrote a wrapper around reqExecutions2() called reqExecutionsFrame(), which converts the list of records into a single data.frame.
So, without further ado, here's the code:
library(IBrokers);
## constructor for an execution object
twsExecution <- function(
execId=NA_character_,
time=NA_character_,
acctNumber=NA_character_,
exchange=NA_character_,
side=NA_character_,
shares=NA_integer_,
price=NA_real_,
permId=NA_integer_,
clientId=NA_integer_, ## no long type in R, but decoding process squeezes longs through ints, so may as well use int
orderId=NA_integer_, ## no long type in R, but decoding process squeezes longs through ints, so may as well use int
liquidation=NA_integer_,
cumQty=NA_integer_,
avgPrice=NA_real_,
orderRef=NA_character_,
evRule=NA_character_,
evMultiplier=NA_real_
) {
structure(
list(
execId=execId,
time=time,
acctNumber=acctNumber,
exchange=exchange,
side=side,
shares=shares,
price=price,
permId=permId,
clientId=clientId,
orderId=orderId,
liquidation=liquidation,
cumQty=cumQty,
avgPrice=avgPrice,
orderRef=orderRef,
evRule=evRule,
evMultiplier=evMultiplier
),
class='twsExecution'
);
}; ## end twsExecution()
print.twsExecution <- function(x,...) str(unclass(x));
## replacement for reqExecutions()
reqExecutions2 <- function(twscon,reqId=0L,filter=list()) {
## validate the connection object
if (!is.twsConnection(twscon)) stop('invalid twsConnection object.');
if (!isConnected(twscon)) stop('peer has gone away. check your IB connection',call.=F);
## shallow validation of args
if (!is.integer(reqId) || length(reqId) != 1L) stop('reqId must be a scalar integer.');
if (!is.list(filter) || (is.null(names(filter)) && length(filter) > 0L)) stop('filter must be a named list.');
## send encoded request
socketcon <- twscon[[1]]; ## extract socket connection from TWS connection object
VERSION <- '3';
prepareField <- function(x) if (is.null(x) || length(x) != 1L || is.na(x)) '' else as.character(x); ## empty string is accepted as unspecified
outgoing <- c(
.twsOutgoingMSG$REQ_EXECUTIONS,
VERSION,
prepareField(reqId), ## will receive this in the response along with data
prepareField(filter$clientId), ## any client id; if invalid, will get zero results
prepareField(filter$acctCode), ## must be a valid account code; seems to be ignored completely if invalid
prepareField(filter$time), ## yyyymmdd HH:MM:SS
prepareField(filter$symbol), ## must be a valid contract symbol, case-insensitive
prepareField(filter$secType), ## STK|OPT|FUT|IND|FOP|CASH|BAG|NEWS
prepareField(filter$exchange), ## must be a valid exchange name, case-insensitive; seems to be ignored completely if invalid
prepareField(filter$side) ## buy|sell
);
writeBin(outgoing,socketcon); ## automatically appends a NUL after each vector element
## set handler method
## note: don't need to explicitly handle execDetailsEnd(); it provides no new data, and the below while-loop will check for and break on it
ew <- eWrapper();
ew$execDetails <- function(curMsg,msg,timestamp,file,...) {
## reqId and most contract and execution fields are returned in a character vector in msg
## build a return value by mapping the fields to their corresponding parameters of twsContract() and twsExecution()
n <- (function() { n <- 0L; function() n <<- n+1L; })();
version <- as.integer(msg[n()]);
reqId <- if (version >= 7L) as.integer(msg[n()]) else -1L;
orderId <- as.integer(msg[n()]); ## not sure why this is out-of-order with the remaining execution fields
## contract fields
conId <- as.integer(msg[n()]);
symbol <- msg[n()];
secType <- msg[n()];
lastTradeDateOrContractMonth <- msg[n()];
strike <- as.double(msg[n()]);
right <- msg[n()];
multiplier <- ''; ##multiplier <- if (version >= 9L) msg[n()] else ''; ----- missing?
exch <- msg[n()];
primaryExchange <- ''; ## not returned
currency <- msg[n()];
localSymbol <- msg[n()];
tradingClass <- if (version >= 10L) msg[n()] else '';
includeExpired <- F; ## not returned
secIdType <- ''; ## not returned
secId <- ''; ## not returned
comboLegsDescrip <- ''; ## not returned
comboLegs <- ''; ## not returned
underComp <- 0L; ## not returned
## execution fields
execId <- msg[n()];
time <- msg[n()];
acctNumber <- msg[n()];
exchange <- msg[n()];
side <- msg[n()];
shares <- as.integer(msg[n()]);
price <- as.double(msg[n()]);
permId <- as.integer(msg[n()]);
clientId <- as.integer(msg[n()]);
## (orderId already assigned)
liquidation <- as.integer(msg[n()]);
cumQty <- if (version >= 6L) as.integer(msg[n()]) else 0L;
avgPrice <- if (version >= 6L) as.double(msg[n()]) else 0;
orderRef <- if (version >= 8L) msg[n()] else '';
evRule <- if (version >= 9L) msg[n()] else '';
evMultiplier <- if (version >= 9L) as.double(msg[n()]) else 0;
## build the list to return
## note: the twsContract() and twsExecution() functions provided with the IBrokers package as of 0.9-12 do not take all of the above fields; we'll pass what they take
list(
reqId=reqId,
contract=twsContract(
conId=conId,
symbol=symbol,
sectype=secType,
exch=exch,
primary=primaryExchange,
expiry=lastTradeDateOrContractMonth,
strike=strike,
currency=currency,
right=right,
local=localSymbol,
multiplier=multiplier,
combo_legs_desc=comboLegsDescrip,
comboleg=comboLegs,
include_expired=includeExpired,
secIdType=secIdType,
secId=secId
),
execution=twsExecution(
execId=execId,
time=time,
acctNumber=acctNumber,
exchange=exchange,
side=side,
shares=shares,
price=price,
permId=permId,
clientId=clientId,
orderId=orderId,
liquidation=liquidation,
cumQty=cumQty,
avgPrice=avgPrice,
orderRef=orderRef,
evRule=evRule,
evMultiplier=evMultiplier
)
);
}; ## end execDetails()
## hack errorMessage() so we can differentiate between true errors and info messages; not the best design on the part of IB to conflate these
body(ew$errorMessage)[[length(body(ew$errorMessage))+1L]] <- substitute(msg);
## iterate until we get the expected responses off the socket
execList <- list();
while (isConnected(twscon)) {
socketSelect(list(socketcon),F,NULL);
curMsg <- readBin(socketcon,character(),1L);
res <- processMsg(curMsg,socketcon,eWrapper=ew,twsconn=twscon,timestamp=NULL,file='');
## check for error
if (curMsg == .twsIncomingMSG$ERR_MSG) {
## note: the actual message was already catted inside processMsg() -> ew$errorMessage(); just abort if true error
code <- as.integer(res[3L]);
if (!code%in%c( ## blacklist info messages
0 , ## "Warning: Approaching max rate of 50 messages per second (%d)"
2103, ## "A market data farm is disconnected."
2104, ## "A market data farm is connected."
2105, ## "A historical data farm is disconnected."
2106, ## "A historical data farm is connected."
2107, ## "A historical data farm connection has become inactive but should be available upon demand."
2108, ## "A market data farm connection has become inactive but should be available upon demand."
2119 ## "Market data farm is connecting:%s" -- undocumented
)) stop(paste0('request error ',code));
}; ## end if
## check for data
if (curMsg == .twsIncomingMSG$EXECUTION_DATA)
execList[[length(execList)+1L]] <- res;
## check for completion
if (curMsg == .twsIncomingMSG$EXECUTION_DATA_END) break;
}; ## end while
execList;
}; ## end reqExecutions2()
reqExecutionsFrame <- function(...) {
res <- reqExecutions2(...);
do.call(rbind,lapply(res,function(e) do.call(data.frame,c(list(reqId=e$reqId),e$contract,e$execution,stringsAsFactors=F))));
}; ## end reqExecutionsFrame()
Here's a demo on my paper trading account:
## create the TWS connection, selecting an arbitrary client id
twscon <- twsConnect(0L);
twscon; ## this is how it displays by default
## <twsConnection,0 @ 20160229 07:36:33 EST, nextId=4268>
(function(x) c(typeof(x),mode(x),class(x)))(twscon); ## show type info
## [1] "environment" "environment" "twsconn" "environment"
ls(twscon); ## list the entries in the environment
## [1] "clientId" "conn" "connected" "connected.at" "nextValidId" "port" "server.version"
twscon$conn; ## actual socket connection across which I/O travels between the client and server
## description class mode text
## "->localhost:7496" "sockconn" "ab" "binary"
## opened can read can write
## "opened" "yes" "yes"
## demo the current time request
## note some info messages are always written onto the socket by the server after we create a connection; the while loop simply passes through them before getting to the current time response
reqCurrentTime(twscon);
## TWS Message: 2 -1 2104 Market data farm connection is OK:cashfarm
## TWS Message: 2 -1 2104 Market data farm connection is OK:usfarm
## TWS Message: 2 -1 2106 HMDS data farm connection is OK:cashhmds
## TWS Message: 2 -1 2106 HMDS data farm connection is OK:ushmds
## [1] "2016-02-29 07:40:10 EST"
## demo the executions request code; shows some random executions I did earlier today (in a paper trading account, of course!)
reqExecutionsFrame(twscon);
## reqId conId symbol sectype exch primary expiry strike currency right local multiplier combo_legs_desc comboleg include_expired secIdType secId execId time acctNumber exchange side shares price permId clientId orderId liquidation cumQty avgPrice orderRef evRule evMultiplier
## 1 0 15016062 USD CASH IDEALPRO 0 CAD USD.CAD FALSE 0001f4e8.56d38c6c.01.01 20160229 02:58:06 XXXXXXXX IDEALPRO SLD 100000 1.35305 195295721 0 2147483647 0 100000 1.35305 <NA> <NA> NA
## 2 0 15016062 USD CASH IDEALPRO 0 CAD USD.CAD FALSE 0001f4e8.56d38c6f.01.01 20160229 02:58:15 XXXXXXXX IDEALPRO BOT 25000 1.35310 195295723 0 2147483647 0 25000 1.35310 <NA> <NA> NA
## 3 0 15016062 USD CASH IDEALPRO 0 CAD USD.CAD FALSE 0001f4e8.56d38c76.01.01 20160229 02:58:42 XXXXXXXX IDEALPRO BOT 75000 1.35330 195295723 0 2147483647 0 100000 1.35325 <NA> <NA> NA
## 4 0 15016062 USD CASH IDEALPRO 0 CAD USD.CAD FALSE 0001f4e8.56d39e0b.01.01 20160229 05:48:50 XXXXXXXX IDEALPRO BOT 100000 1.35710 195295940 0 2147483647 0 100000 1.35710 <NA> <NA> NA
## 5 0 15016062 USD CASH IDEALPRO 0 CAD USD.CAD FALSE 0001f4e8.56d39e16.01.01 20160229 05:49:14 XXXXXXXX IDEALPRO SLD 100000 1.35720 195295942 0 2147483647 0 100000 1.35720 <NA> <NA> NA
## demo some filtering
reqExecutionsFrame(twscon,filter=twsExecutionFilter(side='buy'));
## reqId conId symbol sectype exch primary expiry strike currency right local multiplier combo_legs_desc comboleg include_expired secIdType secId execId time acctNumber exchange side shares price permId clientId orderId liquidation cumQty avgPrice orderRef evRule evMultiplier
## 1 0 15016062 USD CASH IDEALPRO 0 CAD USD.CAD FALSE 0001f4e8.56d38c6f.01.01 20160229 02:58:15 XXXXXXXX IDEALPRO BOT 25000 1.3531 195295723 0 2147483647 0 25000 1.35310 <NA> <NA> NA
## 2 0 15016062 USD CASH IDEALPRO 0 CAD USD.CAD FALSE 0001f4e8.56d38c76.01.01 20160229 02:58:42 XXXXXXXX IDEALPRO BOT 75000 1.3533 195295723 0 2147483647 0 100000 1.35325 <NA> <NA> NA
## 3 0 15016062 USD CASH IDEALPRO 0 CAD USD.CAD FALSE 0001f4e8.56d39e0b.01.01 20160229 05:48:50 XXXXXXXX IDEALPRO BOT 100000 1.3571 195295940 0 2147483647 0 100000 1.35710 <NA> <NA> NA
reqExecutionsFrame(twscon,filter=twsExecutionFilter(side='buy',time='20160229 04:00:00'));
## reqId conId symbol sectype exch primary expiry strike currency right local multiplier combo_legs_desc comboleg include_expired secIdType secId execId time acctNumber exchange side shares price permId clientId orderId liquidation cumQty avgPrice orderRef evRule evMultiplier
## 1 0 15016062 USD CASH IDEALPRO 0 CAD USD.CAD FALSE 0001f4e8.56d39e0b.01.01 20160229 05:48:50 XXXXXXXX IDEALPRO BOT 100000 1.3571 195295940 0 2147483647 0 100000 1.3571 <NA> <NA> NA
## demo error handling
reqExecutionsFrame(twscon,filter=twsExecutionFilter(side='invalid'));
## TWS Message: 2 0 321 Error validating request:-'gf' : cause - Invalid side
## Error in reqExecutions2(...) : request error 321