This is my first question post on StackOverflow, so please correct any protocol errors I make. But I have already benefitted greatly from answers to many other questions on the site, so before I go further I would like to thank the community that contributes to the site.
I am using R, with RStudio, and packages devtools and roxygen2 on a Windows 7 machine (full session_info at the end of the question)
I am trying to write a package of my own, which defines subclasses of the S4 class SpatialLines from the package sp, and also uses the package rgeos. My problem relates to warning messages that I do not really understand, and cannot fully resolve, when I used devtools to document and
load_all my package.
The first message is
class "Spatial" is defined (with package slot ‘sp’) but no metadata object found to revise subclass information---not exported? Making a copy in package ‘minweSpatialNULL’
When I resolve this, by adding an entry in the NAMESPACE file (without fully understanding why it is necessary), and then make one further change to the DESCRIPTION file I get a second message which has me beaten, thus far.
d> devtools::document()
Updating minweSpatialNULL documentation
Loading minweSpatialNULL
Error: class "SpatialLinesNULL" is not exported by 'namespace:rgeos'
I have googled and searched several sites looking for hints, but the closest I found was a brief thread from Dec 2011 asking where SpatialLinesNULL came from --- to which
the answer was rgeos.
http://lists.r-forge.r-project.org/pipermail/rspatial-devel/2011-December/000033.html
I have tried experimenting to understand what is wrong myself, and stepped through code in the debugger till I got lost, way deeper than I could follow well enough to figure out what was supposed to be happening. In the process have produced a small (not) working example of the issue, which is shown below.
I have 3 questions. They are summarised here, but really only make sense in the context of my (not) working toy example below :
Spatial anyway, and why only when I have the 2nd generation MyClass2 defined in my code?@importClassesFromed the class I actually use.rgeos package
for help (eg ask them to export SpatialLinesNULL)?The r code for my toy working example (stripped only of roxygen comments to save space) is :
MyClass1 <- setClass('MyClass1', contains = c('SpatialLines'))
MyClass2 <- setClass('MyClass2', contains = c('MyClass1'))
The MyClass2 2nd generation subclass seems to be significant (without it the warnings go away). I do not understand why.
I have run a sequence of tests on this code, using devtools and
roxygen2. The tests vary by what is in the NAMESPACE file, and what is
in the Imports directive in the DESCRIPTION file. Executing the test consists of running devtools::load_all() (or devtools::document() which calls load_all() along the way) in RStudio. The warning messages which occur have been posted above. I restart my R session in a clean global environment between each test (the warnings only occur the first time I run load_all or document).
While following up an answer given below I have realised I should also have tried to build and install my toy package using R from a shell command line.
test1 has DESCRIPTION file entry (see the end of this question for the complete DESCRIPTION file):
Imports:
methods,
sp
and the complete NAMESPACE file (generated from roxygen comments omitted above) is :
# Generated by roxygen2 (4.1.0.9001): do not edit by hand
importClassesFrom(sp,SpatialLines)
importFrom(sp,SpatialLines)
Trying to run devtools::load_all() (or devtools::document()) in a
restarted R session generates the first warning (shown above).
Question 1 Why does R need the ~~(virtual?)~~ superclass Spatial anyway, and why only when I have the 2nd generation MyClass2 defined in my code? Clarification added I had thought that the R packaging namespace mechanism would have taken care of any need to find and access antecedents / superclasses once I had @importClassesFromed the class I actually use.
test2 resolves the problem by updating the NAMESPACE file as indicated by the warning message. No other changes were made.
The warning goes away when the complete NAMESPACE file is :
# Generated by roxygen2 (4.1.0.9001): do not edit by hand
importClassesFrom(sp,Spatial)
importClassesFrom(sp,SpatialLines)
importFrom(sp,SpatialLines)
An apparently similar problem recurs though, when I made one small change to
the Imports directive in the DESCRIPTION file, namely add in rgeos. This is the only change made to create the test3 scenario from test2.
Imports:
methods,
sp,
rgeos
Now I get a warning that looks like the problem from test1, viz :
class "SpatialLinesNULL" is defined (with package slot ‘rgeos’) but no metadata object found to revise subclass information---not exported? Making a copy in package ‘minweSpatialNULL
As far as I have been able to figure out, SpatialLinesNULL is a new
superclass of Spatial, defined in rgeos so it can provide some
generic functions and methods.
Question 2 I am presuming that whatever the reason is for my first warning message, is also the reason for this similar message.
Am I on the right track?
In test4 I try to apply the same solution as worked in test2, namely add the missing entry to the NAMESPACE file. This is the complete NAMESPACE file for test4.
# Generated by roxygen2 (4.1.0.9001): do not edit by hand
importClassesFrom(rgeos,SpatialLinesNULL)
importClassesFrom(sp,Spatial)
importClassesFrom(sp,SpatialLines)
importFrom(sp,SpatialLines)
at which stage I get my final error message :
d> devtools::document()
Updating minweSpatialNULL documentation
Loading minweSpatialNULL
Error: class "SpatialLinesNULL" is not exported by 'namespace:rgeos
At this stage I am stuck.
Question 3 Is the warning from test3 and this
subsequent error something I can resolve myself (if so how)?
or do I need to ask the maintainer of the rgeos package
for help (eg ask them to export SpatialLinesNULL)?
d> devtools::session_info()
Session info --------------------------------------------------------------------------
setting value
version R version 3.1.2 (2014-10-31)
system x86_64, mingw32
ui RStudio (0.98.953)
language (EN)
collate English_Australia.1252
tz Australia/Sydney
Packages ------------------------------------------------------------------------------
package * version date source
devtools * 1.7.0.9000 2015-02-20 local
lattice * 0.20-29 2014-04-04 CRAN (R 3.1.2)
minweSpatialNULL * 0.0.0.9000 <NA> local
Rcpp * 0.11.4 2015-01-24 CRAN (R 3.1.2)
rgeos * 0.3-8 2014-09-21 CRAN (R 3.1.2)
roxygen2 * 4.1.0.9001 2015-02-21 local
rstudio * 0.98.953 2014-08-02 local
rstudioapi * 0.2 2014-12-31 CRAN (R 3.1.2)
sp * 1.0-17 2015-01-08 CRAN (R 3.1.2)
stringr * 0.6.2 2012-12-06 CRAN (R 3.1.1)
d>
R/myclasses.r
#------------------------------------------------------------------------------
#' MyClass1
#'
#' A subclass of SpatialLines
#'
#' MyClass1 is a subclass of sp::SpatialLines.
#' @importClassesFrom sp SpatialLines Spatial
#' @importFrom sp SpatialLines
#' @importClassesFrom rgeos SpatialLinesNULL
MyClass1 <- setClass('MyClass1', contains = c('SpatialLines'))
#------------------------------------------------------------------------------
#' MyClass2
#'
#' A subclass of MyClass1
#'
#' MyClass2 is a subclass of MyClass1 and a 2nd generation subclass of
#' sp::SpatialLines.
#' @importClassesFrom sp SpatialLines
MyClass2 <- setClass('MyClass2', contains = c('MyClass1'))
DESCRIPTION file
Package: minweSpatialNULL
Title: Minimum Example of my Problem with SpatialLinesNULL
Version: 0.0.0.9000
Authors@R: person("Geoff", "Lee", , "geoff.lee@gmail.com", role = c("aut", "cre"))
Description: (Hopefully) demonstrates my problem with warnings
that I cannot seem to eradicate.
Depends: R (>= 3.1.2)
Imports:
methods,
sp,
rgeos
License: GPL (>= 2.0)
LazyData: true
NAMESPACE file
# Generated by roxygen2 (4.1.0.9001): do not edit by hand
importClassesFrom(rgeos,SpatialLinesNULL)
importClassesFrom(sp,Spatial)
importClassesFrom(sp,SpatialLines)
importFrom(sp,SpatialLines)
Many thanks in advance for any help, advice or guidance that can be offered! Even if it is just about how I might pose a better question :-)
This answer draws together all the information I've learnt related to the suite of questions I asked, for the benefit of anyone else who comes across a similar issue.
It turns out to be easier to present the answers in reverse to the order in which the questions were asked.
The aspect which really resolved my specific problem was provided by @Edzer Pebesma - who has my thanks.
John Chambers answered a more specific follow on question I asked on the r-devel mailing list, and that really helped fill in the gaps in my understanding of why the warning message was being given - I am also very grateful for his guidance and assistance.
Any errors that remain in the explanation below are entirely my own!
The test4 version of the toy package referenced in this question and answer used to be found in --> folder SpatialLinesNULL of branch SpatialLinesNULL in a github repo at http://github.com/Geoff99/Examples/tree/SpatialLinesNULL <-- Other versions of the test sequence can be constructed by deleting a line or two from the NAMESPACE and DESCRIPTION files.
Update 6 May 2017 - test4 version code moved to a gist at https://gist.github.com/Geoff99/29be25bce4cd4c918921bf68769c6a39
Is the warning from test3 and this subsequent error something I can resolve myself (if so how)? or do I need to ask the maintainer of the rgeos package for help (eg ask them to export SpatialLinesNULL)?
The warning from test3 was
class "SpatialLinesNULL" is defined (with package slot ‘rgeos’) but no metadata object found to revise subclass information---not exported? Making a copy in package ‘minweSpatialNULL
and the error message from test4 was
d> devtools::document()
Updating minweSpatialNULL documentation
Loading minweSpatialNULL
Error: class "SpatialLinesNULL" is not exported by 'namespace:rgeos
The best solution to this was to approach a maintainer of the rgeos package and ask them to export the missing SpatialLinesNULL class that is mentioned in the warning and error messages. Edzer Pebesma answered this for me, and also made the necessary updates to the rgeos package, for which I am very grateful. After installing the upgraded version of package rgeos I could simply add importClassesFrom(rgeos,SpatialLinesNULL) to the NAMESPACE file of my toy package, and the warning went away.
If you are facing a similar situation with a different package, that is the strategy I would recommend.
I suspect that there is a way I could have resolved this myself (essentially by manually making a copy of the missing metadata about the SpatialLinesNULL class in the namespace environment of my package, which is basically what the methods package does itself after it issues the warning) --- but it is such a messy hack that I will refrain from explaining how. If you really need to do this at some stage in the future, the answer to question 1 (below) collects the information I needed to understand to figure out how to do the hack.
I am presuming that whatever the reason is for my first warning message, is also the reason for this similar message. Am I on the right track?
The warning from test3 was
class "SpatialLinesNULL" is defined (with package slot ‘rgeos’) but no metadata object found to revise subclass information---not exported? Making a copy in package ‘minweSpatialNULL
and the first warning from test1 was
class "Spatial" is defined (with package slot ‘sp’) but no metadata object found to revise subclass information---not exported? Making a copy in package ‘minweSpatialNULL’
The answer was yes, both messages appear for the same reason.
The warning messages are both being generated by a function called .findOrCopyClass with is an internal (ie non-exported) function in the methods package. To see the code for this function it is necessary to use the ::: operator. Type methods:::.findOrCopyClass into an R console. (This requires the methods package to be loaded, which it almost always is). .findOrCopyClass issues this warning when it cannot find the metadata which defines a class, and so it has to copy the metadata.
See below for what the metadata is, why it needs to be included in the namespace for my toy package, and (my best guess) at why this warning is issued.
.findOrCopyClass is called by setIs, which is an exported function from the methods package. In turn setIs is called (several times) by the methods function setClass, which I call in my toy package example. Type ?setIs and ?setClass for more information about those functions. Or just type setIs or setClass in an R console if you want to see their source code.
Why does R need the ~~(virtual?)~~ superclass Spatial anyway~~, and why only when I have the 2nd generation MyClass2 defined in my code~~?
Clarification added : I had thought that the R packaging namespace mechanism would have taken care of any need to find and access antecedents / superclasses once I had
@importClassesFromed the class I actually use.
The part of the original question that asked:
and why only when I have the 2nd generation MyClass2 defined in my code?
was just plain wrong. Despite my checking before I posted the question, I must have unwittingly still has some hidden information cached in my globalenv(). The underlying issue exists even if I only have MyClass1 defined in my toy package.
Spatial is not a virtual class. SpatialLinesNULL is a virtual class. Whether or not a superclass is virtual has nothing to do with this issue.
Spatial comes from sp and SpatialLinesNULL comes from rgeos)sp and rgeos)minweSpatialNuULL) which uses a class (eg SpatialLines) whose parents include superclasses (eg Spatial, SpatialLinesNULL).The following is an annotated walk through parts of the documentation for the methods package. It took me a while to grasp, but that is to do with my low knowledge base when I began. The documentation is very useful, and if you get this far, I would recommend reading it. I found the entries from ?setClass, ?Classes, ?getClassDef, ?classMetaName and ?setIs particularly valuable.
setClass does
From the ?setClass documentation (with emphasis added by me) :
Create a class definition, specifying the representation (the slots) and/or the classes contained in this one (the superclasses), plus other optional details. As a side effect, the class definition is stored in the specified environment. A generator function is returned as the value of setClass(), suitable for creating objects from the class if the class is not virtual.
The warning message issued by methods:::.findOrCopyClass is related to the creation and storage of the class definition. The class definition is (or is stored in?) a metadata object.
From the ?Classes documentation :
Class definitions are objects that contain the formal definition of a class of R objects, ... and
When a class is defined, an object is stored that contains the information about that class. The object, known as the metadata defining the class, is not stored under the name of the class (to allow programmers to write generating functions of that name), but under a specially constructed name. To examine the class definition, call
getClass. The information in the metadata object includes:
- Slots ...
- SuperClasses ...
- The information about the relation between a class and a particular superclass is encoded as an object of class SClassExtension. A list of such objects for the superclasses (and sometimes for the subclasses) is included in the metadata object defining the class
- Prototype ...
The methods package is all about creating and managing these metadata objects (for S4 classes, and as best as it can, for the older S3 classes as well).
The metadata object for a new class (eg MyClass1) includes information about any slots in its definition (since MyClass1 specifies contains = c('SpatialLines') it inherits all the slots from the SpatialLines class, defined in package sp).
Importantly, the metadata object for a class must also contain information about any superclasses - otherwise how could the new class ever inherit anything from its parents, ie the superclasses ? And one way and another, that information about superclasses must reach all the way up the inheritance tree. In the example, MyClass1 has a distance one superclass SpatialLines, and hence inherits all the more distant ancestors of SpatialLines, whoever they may be.
MyClass1) may not (yet) have been filled in all the way up the inheritance chain, whereas by the time a class definition is used, R has to have traversed the inheritance chain, and filled in the complete genealogy for the class. When a package is checked or installed, the inheritance chain must be complete - if it has not been completed before, it has to be during the install.packages step.Use getClass or getClassDef to see the metadata object for a given class.
The documentation ?getClass or ?getClassDef explains which environments R searches for metadata objects. A key thing to note is that a package must at least be loaded before metadata objects from classes defined in the package can be found.
Here are some examples from my toy package (minweSpatialNULL) after it has been loaded with version 0.3-9 of rgeos, which does export the SpatialLinesNULL class. Notice that:
MyClass1 knows all its parent superclasses (the usual situation), and SpatialLinesNULL (which is a Class Union defined in the rgeos package) knows almost all its children subclasses, even though rgeos itself knows nothing at all about MyClass1 or MyClass2 !
d> getClass('MyClass1')
Class "MyClass1" [package "minweSpatialNULL"]
Slots:
Name: lines bbox proj4string
Class: list matrix CRS
Extends:
Class "SpatialLines", directly
Class "Spatial", by class "SpatialLines", distance 2
Class "SpatialLinesNULL", by class "SpatialLines", distance 2
Known Subclasses: "MyClass2"
d> getClass('SpatialLinesNULL')
Extended class definition ( "ClassUnionRepresentation" )
Virtual Class "SpatialLinesNULL" [package "rgeos"]
No Slots, prototype of class "NULL"
Known Subclasses:
Class "NULL", directly
Class "SpatialLines", directly
Class ".NULL", by class "NULL", distance 2, with explicit coerce
Class "SpatialLinesDataFrame", by class "SpatialLines", distance 2
Class "MyClass2", by class "MyClass1", distance 3
d>
Actually it is slightly curious that SpatialLinesNULL knows about MyClass2 because it is a subclass of MyClass1, but does not appear to mention MyClass1 itself directly. I haven't figured out why yet.
I was puzzled for a while as to why the list of superclasses of my toy MyClass1 class changed when I made no change to my package other than adding rgeos to the Imports directive in the toy package's DESCRIPTION file.
The reason is that the methods package is very clever. As well as defining subclasses, it is possible to insert parent superclasses into the family tree (inheritance chain) of a class, after that class has been defined.
A simple way is to use setClassUnion (type ?setClassUnion for an explanation). That is what rgeos does when it defines SpatialLinesNULL - it creates a new parent for SpatialLines. And that is why as soon as I added the rgeos entry in my toy package DESCRIPTION file, the class MyClass1 suddenly acquired another superclass as well. In an interactive setting something similar would presumably have happened, depending on whether or not I had loaded (or attached and loaded) the rgeos package.
There is a more complicated way to add ancestors after the fact as well - see ?setIs.
As with all other objects, R finds metadata objects via their name. If an object is defined in a package, the name of the object lives in
environment: namespace:minweSpatialNULL), or environment: imports:minweSpatialNULL).From the ?Classes documentation again :
When a class is defined, an object is stored that contains the information about that class. The object, known as the metadata defining the class, is not stored under the name of the class (to allow programmers to write generating functions of that name), but under a specially constructed name.
You can find the specially constructed name using the classMetaName function which the methods package provides - see ?methods::classMetaName for details.
An example of the name of a metadata object, taken from my toy package, is :
d> classMetaName('MyClass1')
[1] ".__C__MyClass1"
Because of the leading . the name of this metadata object (ie .__C__MyClass1) is usually hidden when you ls the namespace environment of the package - but you can see it by using the all.names = TRUE argument to ls.
d> # Recall that MyClass1 is the name I chose for the generator function
d> # returned by setClass when I defined the class MyClass1
d> env_toy_package <- environment(MyClass1)
d> ls(env_toy_package, all.names=TRUE)
[1] ".__C__MyClass1" ".__C__MyClass2" ".__DEVTOOLS__"
[4] ".__NAMESPACE__." ".__S3MethodsTable__." ".packageName"
[7] "MyClass1" "MyClass2"
d>
And if you look in the parent.env of the package namespace environment (ie the imports:minweSpatialNULL namespace environment), you find the hidden names of the superclasses of MyClass1, which is what .findOrCopyClass was looking for so they could be put there!.
d> parent.env(env_toy_package)
<environment: 0x0000000008df59d8>
attr(,"name")
[1] "imports:minweSpatialNULL"
d> ls(parent.env(env_toy_package), all.names = TRUE)
[1] ".__C__Spatial" ".__C__SpatialLines" ".__C__SpatialLinesNULL"
[4] "library.dynam.unload" "SpatialLines" "system.file"
d>
When .findOrCopyClass issues a warning such as :
class "Spatial" is defined (with package slot ‘sp’) but no metadata object found to revise subclass information---not exported? Making a copy in package ‘minweSpatialNULL’
the function is letting me know that it has been unable to find the hidden name .__C__Spatial in the namespaces it has searched and revise the metadata in the object to which that name is bound. It knows where the hidden name should be --- it must come from package sp, because the metadata object from the SpatialLines class itself says where its parent superclass (Spatial) lives. The full specification of an ancestor superclass in a metadata object includes a package attribute containing the name of the package where the ancestor superclass was defined, just in case two existing loaded packages just happen to have chosen the same name for a class!
That is why importClassesFroming the missing superclass works - it brings the missing hidden names into the imports:minweSpatialNULL environment of the minweSpatialNULL toy package, from which spot .findOrCopyClass can find them, and update them if necessary.
.findOrCopy emits a warning
I wondered why .findOrCopyClass told me it couldn't find the metadata object, when the very next thing it said was that it was making a copy in my packages namespace! I posed that question on the r-devel mailing list, and John Chambers kindly answered it for me (emphases added by me):
the purpose of finding the class definition is to update the entry for the new relationship, as the warning message suggests. That requires that the namespace holding the definition be writable.
In the case of subclass information, the original namespace is very likely to be locked, if it's not the package currently being loaded. Copying the definition in order to update subclass information seems the only reasonable choice, and no warning message is needed.
A revised version will omit this message.
My remaining conceptual problem related to the difference between what happens to the name of an object (eg a function or a class (to be precise, a function object or a class metadata object)) and what happens to the object itself. Or in other terms, whether a namespace (possibly locked) needs to be updated, or the (state of the) object itself is what needs to be updated.
It turns out that 'findorCopyClasses is not complaining that it cannot find the (non-exported) superclass, but rather that it cannot find it in a place where the superclass's value can be changed.
The ?base::bindenv documentation says :
" The namespace environments of packages with namespaces are locked when loaded."
and a locked environment means
" prevents adding or removing variable bindings from the environment. Changing the value of a variable is still possible unless the binding has been locked"
But a bit experimentation reveals that as well as package namespace environments being locked, the bindings in the imports:namespace are locked. Locked bindings means that :
" The value of a locked binding cannot be changed".
Hence since the superclass metadata object comes from another package (eg rgeos), its binding in that other package's namespace is locked and hence its value cannot be changed. Or in terms of the example, when the methods package wants to add Myclass1 and MyClass2 to the list of subclasses that SpatialLinesNULL 'owns', it finds it cannot because the binding of SpatialLinesNULL (or to be precise, the binding of the hidden name .__C__SpatialLinesNULL) is locked. Hence the message about needing to make a copy!
An alternative to copying might be to temporarily unlock the binding in the namespace of the otherpackage, update the object, and relock the binding - but since I have only just learnt about locking environments and bindings, I have no idea what the consequences of that might be. I'll leave well enough alone.
OK, How to hack a solution if the required superclass is not exportedCaveat I haven't tested this much, and it's a bad idea anyway, but if you are really really desperate to get the warning message to go away ...
Step1. Look at the warning message and find the (text) name of the missing superclass, and the package it comes from.
Step2. Use classMetaName to find the hidden and mangled name of the missing unexported superclass. For superclass MissingSuperclass it will probably be .__C__MissingSuperclass
Step3. Use the ::: approach to access the metadata object (ie donorpackage:::.__C__MissingSuperclass) and give it the appropriate name (ie .__C__MissingSuperclass) in the appropriate environment belonging to your package, before calling setClass to create your own class.
As I've already said, this is a really bad idea, since it would be enormously easy to shoot yourself in the foot, and why bother anyway, since that's essentially what .findOrCopyClass and setIs seem to do after the warning message has been sent anyway.
If you've read this far, I hope this has helped you! I've written it at such length mostly as a tutorial for a future me :-)